Introduction¶

Hummingbird (the bird): Hummingbirds are among the smallest of birds. They hover in mid-air at rapid wing flapping rates, typically around 50 times per second, but possibly as high as 200 times per second, allowing them also to fly at speeds exceeding 15 m/s (54 km/h; 34 mph), backwards or upside down. [..]. (Wikipedia).

Hummingbird provides a Web Processing Service (WPS) using PyWPS for the climate science community. It has WPS processes for common tools used in climate science like CDO and cfchecker.

Hummingbird is part of the Birdhouse project.

Contents:

Installation¶

Check out code from the Hummingbird github repo and start the installation:

$ git clone https://github.com/bird-house/hummingbird.git
$ cd hummingbird
$ make clean install

For other install options run make help and read the documention of the Makefile.

After successful installation you need to start the services. Hummingbird is using Anaconda Python distribution system. All installed files (config etc ...) are below the Anaconda root folder which is by default in your home directory ~/anaconda. Now, start the services:

$ make start    # starts supervisor services
$ make status   # shows supervisor status

The depolyed WPS service is by default available on http://localhost:8092/wps?service=WPS&version=1.0.0&request=GetCapabilities.

Check the log files for errors:

$ tail -f  ~/birdhouse/var/log/pywps/hummingbird.log
$ tail -f  ~/birdhouse/var/log/supervisor/hummingbird.log

Using docker-compose¶

Start hummingbird with docker-compose (docker-compose version > 1.7):

$ docker-compose up

By default the WPS is available on port 8080: http://localhost:8080/wps?service=WPS&version=1.0.0&request=GetCapabilities.

You can change the ports and hostname with environment variables:

$ HOSTNAME=hummingbird HTTP_PORT=8092 SUPERVISOR_PORT=48092 docker-compose up

Now the WPS is available on port 8092: http://hummingbird:8092/wps?service=WPS&version=1.0.0&request=GetCapabilities.

Configuration¶

If you want to run on a different hostname or port then change the default values in custom.cfg:

$ cd hummingbird
$ vim custom.cfg
$ cat custom.cfg
[settings]
hostname = localhost
http-port = 8092

After any change to your custom.cfg you need to run make update again and restart the supervisor service:

$ make update    # or install
$ make restart

Running unit tests¶

Run quick tests:

$ make test

Run all tests (slow, online):

$ make testall

Check pep8:

$ make pep8

WPS Processes¶

We describe here the WPS processes available in Hummingbird.

WPS Capabilities¶

Using the default Hummingbird installation the GetCapabilities request is as follows:

http://localhost:8092/wps?service=WPS&version=1.0.0&request=GetCapabilities

The XML response of the WPS service is the following document:

<?xml version="1.0" encoding="utf-8"?>
<wps:Capabilities service="WPS" version="1.0.0" xml:lang="en-CA" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsGetCapabilities_response.xsd" updateSequence="1">
	<ows:ServiceIdentification>
		<ows:Title>Hummingbird</ows:Title>
		<ows:Abstract>WPS processes for general tools used in the climate science community like cdo</ows:Abstract>
		<ows:Keywords>
			<ows:Keyword>WPS</ows:Keyword>
			<ows:Keyword>PyWPS</ows:Keyword>
		</ows:Keywords>
		<ows:ServiceType>WPS</ows:ServiceType>
		<ows:ServiceTypeVersion>1.0.0</ows:ServiceTypeVersion>
		<ows:Fees>None</ows:Fees>
		<ows:AccessConstraints>None</ows:AccessConstraints>
	</ows:ServiceIdentification>
	<ows:ServiceProvider>
		<ows:ProviderName></ows:ProviderName>
		<ows:ServiceContact>
			<ows:ContactInfo>
				<ows:HoursOfService>0:00-24:00</ows:HoursOfService>
				<ows:ContactInstructions>None</ows:ContactInstructions>
			</ows:ContactInfo>
		</ows:ServiceContact>
	</ows:ServiceProvider>
	<ows:OperationsMetadata>
		<ows:Operation name="GetCapabilities">
			<ows:DCP>
				<ows:HTTP>
					<ows:Get xlink:href="http://localhost:8092/wps?"/>
					<ows:Post xlink:href="http://localhost:8092/wps"/>
				</ows:HTTP>
			</ows:DCP>
		</ows:Operation>
		<ows:Operation name="DescribeProcess">
			<ows:DCP>
				<ows:HTTP>
					<ows:Get xlink:href="http://localhost:8092/wps?"/>
					<ows:Post xlink:href="http://localhost:8092/wps"/>
				</ows:HTTP>
			</ows:DCP>
		</ows:Operation>
		<ows:Operation name="Execute">
			<ows:DCP>
				<ows:HTTP>
					<ows:Get xlink:href="http://localhost:8092/wps?"/>
					<ows:Post xlink:href="http://localhost:8092/wps"/>
				</ows:HTTP>
			</ows:DCP>
		</ows:Operation>
	</ows:OperationsMetadata>
	<wps:ProcessOfferings>
		<wps:Process wps:processVersion="0.1">
			<ows:Identifier>cdo_sinfo</ows:Identifier>
			<ows:Title>CDO sinfo</ows:Title>
			<ows:Abstract>Apply CDO sinfo on NetCDF File.</ows:Abstract>
                        <ows:Metadata xlink:title="CDO" xlink:href="https://code.zmaw.de/projects/cdo" />
		</wps:Process>
		<wps:Process wps:processVersion="0.1">
			<ows:Identifier>cdo_operation</ows:Identifier>
			<ows:Title>CDO Operation</ows:Title>
			<ows:Abstract>Apply CDO Operation like monmax on NetCDF File.</ows:Abstract>
                        <ows:Metadata xlink:title="CDO" xlink:href="https://code.zmaw.de/projects/cdo" />
		</wps:Process>
		<wps:Process wps:processVersion="0.1">
			<ows:Identifier>cfchecker</ows:Identifier>
			<ows:Title>CF Checker</ows:Title>
			<ows:Abstract>The cfchecker checks NetCDF files for compliance to the CF standard.</ows:Abstract>
		</wps:Process>
	</wps:ProcessOfferings>
	<wps:Languages>
		<wps:Default>
			<ows:Language>en-CA</ows:Language>
		</wps:Default>
		<wps:Supported>
			<ows:Language>en-CA</ows:Language>
		</wps:Supported>
	</wps:Languages>
	<wps:WSDL xlink:href="http://localhost:8092/wps?WSDL"/>
</wps:Capabilities>

SpotChecker¶

Spot Checker checks a single dataset (NetCDF or OpenDAP) against a variety of compliance standards. Available compliance standards are the Climate and Forecast conventions (CF) and project specific rules for CMIP6 and CORDEX.

CFChecker¶

The cfchecker checks NetCDF files for compliance to the Climate Forcast Conventions (CF) standard.

The process expects one or more NetCDF files which should be checked and an optional parameter for the CF version.

WPS process description¶

Using the default Hummingbird installation the DescribeProcess request is as follows:

http://localhost:8092/wps?service=WPS&version=1.0.0&request=DescribeProcess&identifier=cfchecker

The XML response of the WPS service is the following document:

<?xml version="1.0" encoding="utf-8"?>
<wps:ProcessDescriptions xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsDescribeProcess_response.xsd" service="WPS" version="1.0.0" xml:lang="en-CA">
    <ProcessDescription wps:processVersion="0.1" storeSupported="true" statusSupported="true">
        <ows:Identifier>cfchecker</ows:Identifier>
        <ows:Title>CF Checker</ows:Title>
        <ows:Abstract>The cfchecker checks NetCDF files for compliance to the CF standard.</ows:Abstract>
        <DataInputs>
            <Input minOccurs="1" maxOccurs="1000">
                <ows:Identifier>resource</ows:Identifier>
                <ows:Title>NetCDF File</ows:Title>
                <ows:Abstract>NetCDF File</ows:Abstract>
                <ComplexData>
                    <Default>
                        <Format>
                            <MimeType>application/x-netcdf</MimeType>
                        </Format>
                    </Default>
                    <Supported>
                        <Format>
                            <MimeType>application/x-netcdf</MimeType>
                        </Format>
                    </Supported>
                </ComplexData>
            </Input>
            <Input minOccurs="0" maxOccurs="1">
                <ows:Identifier>cf_version</ows:Identifier>
                <ows:Title>CF version</ows:Title>
                <ows:Abstract>CF version to check against, use auto to auto-detect the file version.</ows:Abstract>
                <LiteralData>
                    <ows:DataType ows:reference="http://www.w3.org/TR/xmlschema-2/#string">string</ows:DataType>
                    <ows:AllowedValues>
                            <ows:Value>auto</ows:Value>
                            <ows:Value>1.6</ows:Value>
                            <ows:Value>1.5</ows:Value>
                            <ows:Value>1.4</ows:Value>
                            <ows:Value>1.3</ows:Value>
                            <ows:Value>1.2</ows:Value>
                            <ows:Value>1.1</ows:Value>
                    </ows:AllowedValues>
                    <DefaultValue>auto</DefaultValue>
                </LiteralData>
            </Input>
        </DataInputs>
        <ProcessOutputs>
            <Output>
                <ows:Identifier>output</ows:Identifier>
                <ows:Title>CF Checker Report</ows:Title>
                <ComplexOutput>
                    <Default>
                        <Format>
                            <MimeType>text/plain</MimeType>
                        </Format>
                    </Default>
                    <Supported>
                        <Format>
                            <MimeType>text/plain</MimeType>
                        </Format>
                    </Supported>
                </ComplexOutput>
            </Output>
        </ProcessOutputs>
    </ProcessDescription>
</wps:ProcessDescriptions>

The WPS Parameters are:

resource: Is the input parameter to provide one or more URLs (http://, file://) to NetCDF files. It is a WPS ComplexData type with MIME-type application/x-netcdf.
cf_version: Is an optional input parameter to provide the CF version to check against. It is a WPS LiteralData type with a set of allowed values (1.1, 1.2, ..., auto).
output: Is the output parameter to provide the report of the CF check as text document. It is a WPS ComplexData type with MIME-type text/plain.

WPS process execution¶

An example execution of the cfchecker process with public available data:

http://localhost:8092/wps?service=WPS&version=1.0.0&request=Execute&identifier=cfchecker&DataInputs=resource=http://www.esrl.noaa.gov/psd/thredds/fileServer/Datasets/ncep/vwnd.sfc.2015.nc&RawDataOutput=output

The process is called with key/value parameters, synchronously and with direct output (RawDataOutput).

The resulting text document of the cfchecker report looks like the following:

CDO¶

to be continued ...

Example: Using Docker¶

If you just want to try the Hummingbird Web Processing Service you can also use the Docker image:

$ docker run -i -d -p 9001:9001 -p 8000:8000 -p 8080:8080 birdhouse/hummingbird

Open your browser and enter the url of the supervisor service:

http://localhost:9001/

Run a GetCapabilites WPS request:

http://localhost:8080/wps?service=WPS&version=1.0.0&request=getcapabilities

Run DescribeProcess WPS request for CFChecker:

http://localhost:8080/wps?service=WPS&version=1.0.0&request=describeprocess&identifier=cfchecker

Execute CFChecker process with public available data:

http://localhost:8080/wps?service=WPS&version=1.0.0&request=Execute&identifier=cfchecker&DataInputs=dataset=http://www.esrl.noaa.gov/psd/thredds/fileServer/Datasets/ncep/vwnd.sfc.2015.nc&RawDataOutput=output

Install Birdy WPS command line tool from Anaconda (Anaconda needs to be installed and in your PATH):

$ conda install -c birdhouse birdhouse-birdy

Use Birdy to access Hummingbird WPS service:

$ export WPS_SERVICE=http://localhost:8080/wps
$ birdy -h
$ birdy cfchecker -h
$ birdy cfchecker --dataset http://www.esrl.noaa.gov/psd/thredds/fileServer/Datasets/ncep/vwnd.sfc.2015.nc

Sphinx AutoAPI Index¶

This page is the top-level of your generated API documentation. Below is a list of all items that are documented here.

hummingbird¶

hummingbird.application(environ, start_response)¶

hummingbird.config¶

hummingbird.config.cache_path()¶

hummingbird.exceptions¶

ProcessError¶

class hummingbird.exceptions.ProcessError¶

Imports

exceptions.Exception

hummingbird.processes¶

hummingbird.processes.wps_cdo_bbox¶

Summary

Processes with cdo commands

CDOBBox¶

class hummingbird.processes.wps_cdo_bbox.CDOBBox¶

Imports

<UNKNOWN>

Summary

This process calls cdo sellonlatbox on netcdf file

_handler(self, request, response)¶

hummingbird.processes.wps_cdo_ensembles¶

Summary

Processes with cdo ensemble opertions

CDOEnsembles¶

class hummingbird.processes.wps_cdo_ensembles.CDOEnsembles¶

Imports

<UNKNOWN>

_handler(self, request, response)¶

hummingbird.processes.wps_cdo_indices¶

Summary

Processes with cdo commands

CDOClimateIndices¶

class hummingbird.processes.wps_cdo_indices.CDOClimateIndices¶

Imports

<UNKNOWN>

Summary

This process calls cdo to calculate climate indices written to a netcdf file

_handler(self, request, response)¶

hummingbird.processes.wps_cdo_op¶

Summary

Processes with cdo commands

CDOOperation¶

class hummingbird.processes.wps_cdo_op.CDOOperation¶

Imports

<UNKNOWN>

Summary

This process calls cdo with operation on netcdf file

_handler(self, request, response)¶

hummingbird.processes.wps_cdo_sinfo¶

Summary

Processes with cdo commands

CDOInfo¶

class hummingbird.processes.wps_cdo_sinfo.CDOInfo¶

Imports

<UNKNOWN>

Summary

This process calls cdo sinfo on netcdf file

_handler(self, request, response)¶

hummingbird.processes.wps_cfchecker¶

hummingbird.processes.wps_cfchecker.cf_check(nc_file, version)¶

CFChecker¶

class hummingbird.processes.wps_cfchecker.CFChecker¶

Imports

<UNKNOWN>

_handler(self, request, response)¶

hummingbird.processes.wps_cmor_checker¶

CMORChecker¶

class hummingbird.processes.wps_cmor_checker.CMORChecker¶

Imports

<UNKNOWN>

_handler(self, request, response)¶

hummingbird.processes.wps_compliance_checker¶

CChecker¶

class hummingbird.processes.wps_compliance_checker.CChecker¶

Imports

<UNKNOWN>

_handler(self, request, response)¶

hummingbird.processes.wps_hdh_cfchecker¶

HDHCFChecker¶

class hummingbird.processes.wps_hdh_cfchecker.HDHCFChecker¶

Imports

<UNKNOWN>

_handler(self, request, response)¶

hummingbird.processes.wps_hdh_qachecker¶

QualityChecker¶

class hummingbird.processes.wps_hdh_qachecker.QualityChecker¶

Imports

<UNKNOWN>

_handler(self, request, response)¶

hummingbird.processes.wps_ncdump¶

NCDump¶

class hummingbird.processes.wps_ncdump.NCDump¶

Imports

<UNKNOWN>

_handler(self, request, response)¶

hummingbird.processes.wps_ncplot¶

Summary

Processes for plotting netcdf files with matplotlib/basemap

SimplePlot¶

class hummingbird.processes.wps_ncplot.SimplePlot¶

Imports

<UNKNOWN>

Summary

Plots a simple 2D map of netcdf file

execute(self)¶

hummingbird.processes.wps_spotchecker¶

SpotChecker¶

class hummingbird.processes.wps_spotchecker.SpotChecker¶

Imports

<UNKNOWN>

_handler(self, request, response)¶

hummingbird.processing¶

hummingbird.processing.cmor_checker(dataset, cmip6_table)¶

hummingbird.processing.cmor_dump_output(dataset, status, output, output_filename)¶

hummingbird.processing.cmor_tables()¶

hummingbird.processing.cmor_tables_path()¶

hummingbird.processing.hdh_cf_check(filename)¶

hummingbird.processing.hdh_qa_checker(filename, project)¶

hummingbird.processing.ncdump(dataset)¶

Returns the metadata of the dataset

Code taken from https://github.com/ioos/compliance-checker-web

hummingbird.tests¶

WpsTestCase¶

class hummingbird.tests.WpsTestCase¶

Imports

<UNKNOWN>

Summary

Base TestCase class, sets up a wps

setUpClass(cls)¶

hummingbird.tests.common¶

hummingbird.tests.common.client_for(service)¶

WpsTestClient¶

class hummingbird.tests.common.WpsTestClient¶

Imports

<UNKNOWN>

get(self)¶

hummingbird.tests.test_wps_caps¶

hummingbird.tests.test_wps_caps.test_wps_caps()¶

hummingbird.tests.test_wps_cdo_ensembles¶

hummingbird.tests.test_wps_cdo_ensembles.test_wps_ensembles()¶

hummingbird.tests.test_wps_cdo_op¶

hummingbird.tests.test_wps_cdo_op.test_wps_cdo_operation()¶

hummingbird.tests.test_wps_cdo_sinfo¶

hummingbird.tests.test_wps_cdo_sinfo.test_wps_cdo_sinfo()¶

hummingbird.tests.test_wps_cfchecker¶

hummingbird.tests.test_wps_cfchecker.test_wps_cfchecker()¶

hummingbird.tests.test_wps_compliance_checker¶

hummingbird.tests.test_wps_compliance_checker.test_wps_cchecker()¶

hummingbird.tests.test_wps_hdh¶

hummingbird.tests.test_wps_hdh.test_wps_qa_cfchecker()¶

hummingbird.tests.test_wps_ncdump¶

hummingbird.tests.test_wps_ncdump.test_wps_ncdump()¶

hummingbird.utils¶

hummingbird.utils.fix_filename(filename)¶

hummingbird.utils.make_dirs(path)¶

Introduction¶

Installation¶

Using docker-compose¶

Configuration¶

Running unit tests¶

WPS Processes¶

WPS Capabilities¶

SpotChecker¶

CFChecker¶

WPS process description¶

WPS process execution¶

CDO¶

Example: Using Docker¶

Sphinx AutoAPI Index¶

hummingbird¶

hummingbird.config¶

hummingbird.exceptions¶

ProcessError¶

hummingbird.processes¶

hummingbird.processes.wps_cdo_bbox¶

CDOBBox¶

hummingbird.processes.wps_cdo_ensembles¶

CDOEnsembles¶

hummingbird.processes.wps_cdo_indices¶

CDOClimateIndices¶

hummingbird.processes.wps_cdo_op¶

CDOOperation¶

hummingbird.processes.wps_cdo_sinfo¶

CDOInfo¶

hummingbird.processes.wps_cfchecker¶

CFChecker¶

hummingbird.processes.wps_cmor_checker¶

CMORChecker¶

hummingbird.processes.wps_compliance_checker¶

CChecker¶

hummingbird.processes.wps_hdh_cfchecker¶

HDHCFChecker¶

hummingbird.processes.wps_hdh_qachecker¶

QualityChecker¶

hummingbird.processes.wps_ncdump¶

NCDump¶

hummingbird.processes.wps_ncplot¶

SimplePlot¶

hummingbird.processes.wps_spotchecker¶

SpotChecker¶

hummingbird.processing¶

hummingbird.tests¶

WpsTestCase¶

hummingbird.tests.common¶

WpsTestClient¶

hummingbird.tests.test_wps_caps¶

hummingbird.tests.test_wps_cdo_ensembles¶

hummingbird.tests.test_wps_cdo_op¶

hummingbird.tests.test_wps_cdo_sinfo¶

hummingbird.tests.test_wps_cfchecker¶

hummingbird.tests.test_wps_compliance_checker¶

hummingbird.tests.test_wps_hdh¶

hummingbird.tests.test_wps_ncdump¶

hummingbird.utils¶

Indices and tables¶