-
Notifications
You must be signed in to change notification settings - Fork 9
Using Obspy to load data into PH5
PH5 natively supports common data loggers in use by the IRIS PASSCAL facility. These include reftek 125a and 130, Q330, Geodes, and Fairfield 1 channel and 3 channel nodal instruments. As has grown it has become important to allow community members to easily import their own data in to PH5.
To accomplish this, two new packages were developed: metadatatoph5 and obspytoph5. Using these two packages, allows for the easy ingestion of new data types.
The easiest way to ingest metadata in to PH5 for a custom data source is to use stationXML. Creating stationXML allows you to load all the necessary metadata and response information into PH5 at one time.
A few things to note: PH5 works based on datalogger serial number, so in your stationxml a datalogger serial number will be required in order to load waveform data into PH5. Second, PH5 fundamentally works at the channel level. In your stationXML it is advisable to put lat, lon, and elevation for each channel. Metadatatoph5 though will use assume station lat, lon and elevation for associated channels in the case channel locations are not present.
PASSCAL provides a tool called Nexus that can be used to edit stationXML. Nexus is also capable of grabbing RESP files from the IRIS Nominal Response Library and calculating responses if your instrumentation is in the NRL.
Once you have a valid stationXML file describing the data you would like to load, it can be loaded using metadatatoph5 from the command line metadatatoph5 -n master.ph5 -f <file-name>. If you do not have a PH5 archive master.ph5 yet, this program will create one for you.
As an example we will load a simple stationxml file describing a single station and channel in to PH5. This PH5 loaded with metadata will be used later in the tutorial to load custom waveform data described by this stationxml.
<?xml version='1.0' encoding='UTF-8'?>
<FDSNStationXML xmlns="http://www.fdsn.org/xml/station/1" schemaVersion="1.0">
<Source/>
<Module>Nexus.2018.142.beta</Module>
<ModuleURI>www.passcal.nmt.edu</ModuleURI>
<Created>2019-03-14T17:17:10.202178Z</Created>
<Network code="XX" endDate="2019-03-14T23:59:00.000000Z" startDate="2019-03-14T00:00:00.000000Z">
<Station code="12345" endDate="2019-03-14T23:59:00.000000Z" startDate="2019-03-14T00:00:00.000000Z">
<Latitude unit="DEGREES">0.0</Latitude>
<Longitude unit="DEGREES">0.0</Longitude>
<Elevation unit="METERS">0.0</Elevation>
<Site>
<Name></Name>
</Site>
<CreationDate>2019-03-14T17:17:48.633005Z</CreationDate>
<Channel code="DPZ" endDate="2019-03-14T05:01:00.000000Z" locationCode="01" startDate="2019-03-14T05:00:00.000000Z">
<Comment>
<Value></Value>
</Comment>
<Latitude unit="DEGREES">34.054673</Latitude>
<Longitude unit="DEGREES">-106.906169</Longitude>
<Elevation unit="METERS">1400.0</Elevation>
<Depth unit="METERS">0.0</Depth>
<Azimuth unit="DEGREES">0.0</Azimuth>
<Dip unit="DEGREES">90.0</Dip>
<SampleRate>250.0</SampleRate>
<Sensor>
<Description></Description>
</Sensor>
<DataLogger>
<SerialNumber>11111</SerialNumber>
</DataLogger>
</Channel>
</Station>
</Network>
</FDSNStationXML>With this file saved to disk we load it with the following command:
metadatatoph5 -n master.ph5 -f example.xml
We should get an output like this:
metadatatoph5 -n master.ph5 -f example.xml
[2019-03-14 17:24:57,730] - ph5.utilities.metadatatoph5 - WARNING: ./master.ph5 not found. Creating...
[2019-03-14 17:24:57,766] - ph5.utilities.initialize_ph5 - INFO: Creating temporary receiver_t.tmp kef file using default values.
[2019-03-14 17:24:57,770] - ph5.utilities.initialize_ph5 - INFO: Loading Experiment_g/Receivers_g/Receiver_t using receiver_t.tmp.
[2019-03-14 17:24:57,773] - ph5.utilities.metadatatoph5 - INFO: Removing temporary receiver_t.tmp kef file.
[2019-03-14 17:24:57,779] - ph5.utilities.metadatatoph5 - INFO: Done... Created new PH5 file ./master.ph5.
[2019-03-14 17:24:57,816] - ph5.utilities.metadatatoph5 - INFO: File example.xml is STATIONXML...
[2019-03-14 17:24:57,816] - ph5.utilities.metadatatoph5 - INFO: *****************
[2019-03-14 17:24:57,818] - ph5.utilities.metadatatoph5 - INFO: Found station 12345
[2019-03-14 17:24:57,820] - ph5.utilities.metadatatoph5 - INFO: Found channel DPZ
[2019-03-14 17:24:57,821] - ph5.utilities.metadatatoph5 - INFO: Loaded channel DPZ
[2019-03-14 17:24:57,821] - ph5.utilities.metadatatoph5 - INFO: Loaded Station 12345
[2019-03-14 17:24:57,822] - ph5.utilities.metadatatoph5 - INFO: ******************
Our station and channel metadata is now loaded. To verify we can run: tabletokef -n master.ph5 -A 1
Giving us an output similar to:
tabletokef -n master.ph5 -A 1
#
# Thu Mar 14 17:27:20 2019 ph5 version: 4.1.2
#
# Table row 1
/Experiment_g/Sorts_g/Array_t_001
id_s=12345
location/X/value_d=-106.906169
location/X/units_s=degrees
location/Y/value_d=34.054673
location/Y/units_s=degrees
location/Z/value_d=1400.0
location/Z/units_s=m
location/coordinate_system_s=
location/projection_s=
location/ellipsoid_s=
location/description_s=
deploy_time/ascii_s=2019-03-14T05:00:00
deploy_time/epoch_l=1552539600
deploy_time/micro_seconds_i=0
deploy_time/type_s=BOTH
pickup_time/ascii_s=2019-03-14T05:01:00
pickup_time/epoch_l=1552539660
pickup_time/micro_seconds_i=0
pickup_time/type_s=BOTH
das/serial_number_s=11111
das/model_s=None
das/manufacturer_s=None
das/notes_s=None
sensor/serial_number_s=None
sensor/model_s=None
sensor/manufacturer_s=None
sensor/notes_s=None
description_s=
seed_band_code_s=D
sample_rate_i=250
sample_rate_multiplier_i=1
seed_instrument_code_s=P
seed_orientation_code_s=Z
seed_location_code_s=01
seed_station_name_s=12345
channel_number_i=3
receiver_table_n_i=0
response_table_n_i=0
metadatatoph5 can also be used as a module in your own applications to load an obspy inventory object in to PH5. An example on how to do this will be provided here soon
Using the obspytoph5 package in your own software allows you to load obspy streams into PH5, provided the metadata already exists in PH5. Below will show a simple example of creating an obspy stream with data from a text file and loading it in to PH5. In this example our data is just a text file of random data with one sample per line. Example data