EMTF XML

Anna Kelbert from the USGS has developed an XML schema to contain transfer functions and metadata EMTF XML. This format is currently the working format for EM transfer functions archived at IRIS. This format is the most flexible, accommodating, self-describing, and machine readable format to date.

XML is not the easiest format to look at and manipulate. Here we have tried to make this easier by providing convenience methods for reading, writing, and manipulating these files through the TF class.

[1]:

from mt_metadata import TF_XML
from mt_metadata.transfer_functions import TF
from mt_metadata.transfer_functions.io.emtfxml import EMTFXML

EMTF XML file

These are not common files yet, so let’s print the metadata in an EMTF XML file. It can be verbose, but that’s what makes it an accommodating format.

[2]:

with open(TF_XML, "r") as fid:
    lines = fid.readlines()
print("".join(lines[0:234]))

<?xml version="1.0" encoding="UTF-8"?>
<EM_TF>
  <Description>Magnetotelluric Transfer Functions</Description>
  <ProductId>USMTArray.NMX20.2020</ProductId>
  <SubType>MT_TF</SubType>
  <Notes/>
  <Tags>impedance,tipper</Tags>
  <ExternalUrl>
    <Description>IRIS DMC MetaData</Description>
    <Url>http://www.iris.edu/mda/ZU/NMX20</Url>
  </ExternalUrl>
  <PrimaryData>
    <Filename>NMX20b_NMX20_NMW20_COR21_NMY21-NMX20b_NMX20_UTS18.png</Filename>
  </PrimaryData>
  <Attachment>
    <Filename>NMX20b_NMX20_NMW20_COR21_NMY21-NMX20b_NMX20_UTS18.zmm</Filename>
    <Description>The original used to produce the XML</Description>
  </Attachment>
  <Provenance>
    <CreateTime>2021-03-17T14:47:44</CreateTime>
    <CreatingApplication>EMTF File Conversion Utilities 4.0</CreatingApplication>
    <Creator>
      <Name>Jade Crosbie, Paul Bedrosian and Anna Kelbert</Name>
      <Email>pbedrosian@usgs.gov</Email>
      <Org>U.S. Geological Survey</Org>
      <OrgUrl>https://www.usgs.gov/natural-hazards/geomagnetism</OrgUrl>
    </Creator>
    <Submitter>
      <Name>Anna Kelbert</Name>
      <Email>akelbert@usgs.gov</Email>
      <Org>U.S. Geological Survey, Geomagnetism Program</Org>
      <OrgUrl>https://www.usgs.gov/natural-hazards/geomagnetism</OrgUrl>
    </Submitter>
  </Provenance>
  <Copyright>
    <Citation>
      <Title>USMTArray South Magnetotelluric Transfer Functions</Title>
      <Authors>Schultz, A., Pellerin, L., Bedrosian, P., Kelbert, A., Crosbie, J.</Authors>
      <Year>2020-2023</Year>
      <SurveyDOI>doi:10.17611/DP/EMTF/USMTARRAY/SOUTH</SurveyDOI>
    </Citation>
    <Acknowledgement>The USMTArray-CONUS South campaign was carried out through a cooperative agreement between
the U.S. Geological Survey (USGS) and Oregon State University (OSU). A subset of 40 stations
in the SW US were funded through NASA grant 80NSSC19K0232.
Land permitting, data acquisition, quality control and field processing were
carried out by Green Geophysics with project management and instrument/engineering
support from OSU and Chaytus Engineering, respectively.
Program oversight, definitive data processing and data archiving were provided
by the USGS Geomagnetism Program and the Geology, Geophysics and Geochemistry Science Centers.
We thank the U.S. Forest Service, the Bureau of Land Management, the National Park Service,
the Department of Defense, numerous state land offices and the many private landowners
who permitted land access to acquire the USMTArray data.</Acknowledgement>
    <ReleaseStatus>Unrestricted Release</ReleaseStatus>
    <ConditionsOfUse> All data and metadata for this survey are available free of charge and may be copied freely, duplicated and further distributed provided that this data set is cited as the reference, and that the author(s) contributions are acknowledged as detailed in the Acknowledgements. Any papers cited in this file are only for reference. There is no requirement to cite these papers when the data are used. Whenever possible, we ask that the author(s) are notified prior to any publication that makes use of these data.
 While the author(s) strive to provide data and metadata of best possible quality, neither the author(s) of this data set, nor IRIS make any claims, promises, or guarantees about the accuracy, completeness, or adequacy of this information, and expressly disclaim liability for errors and omissions in the contents of this file. Guidelines about the quality or limitations of the data and metadata, as obtained from the author(s), are included for informational purposes only.</ConditionsOfUse>
  </Copyright>
  <Site>
    <Project>USMTArray</Project>
    <Survey>CONUS South</Survey>
    <YearCollected>2020</YearCollected>
    <Country>USA</Country>
    <Id>NMX20</Id>
    <Name>Nations Draw, NM, USA</Name>
    <Location datum="WGS84">
      <Latitude>34.470528</Latitude>
      <Longitude>-108.712288</Longitude>
      <Elevation units="meters">1940.050</Elevation>
      <Declination epoch="2020.0">9.090</Declination>
    </Location>
    <Orientation angle_to_geographic_north="0.000">orthogonal</Orientation>
    <AcquiredBy>National Geoelectromagnetic Facility</AcquiredBy>
    <Start>2020-09-20T19:03:06</Start>
    <End>2020-10-07T20:28:00</End>
    <RunList>NMX20a NMX20b</RunList>
    <DataQualityNotes>
      <Rating>5</Rating>
      <GoodFromPeriod>5.000</GoodFromPeriod>
      <GoodToPeriod>29127.000</GoodToPeriod>
      <Comments author="Jade Crosbie, Paul Bedrosian and Anna Kelbert">great TF from 10 to 10000 secs (or longer)</Comments>
    </DataQualityNotes>
  </Site>
  <FieldNotes run="NMX20a">
    <Instrument>
      <Manufacturer>Barry Narod</Manufacturer>
      <Name>NIMS</Name>
      <Id>2612-01</Id>
      <Settings/>
    </Instrument>
    <Magnetometer type="fluxgate">
      <Manufacturer>Barry Narod</Manufacturer>
      <Name>NIMS</Name>
      <Id>2509-23</Id>
      <Settings/>
    </Magnetometer>
    <Dipole name="Ex" type="wire">
      <Manufacturer>Oregon State University</Manufacturer>
      <Length units="meters">100.000</Length>
      <Azimuth units="degrees">9.063</Azimuth>
      <Electrode location="N" number="40201038">Pb-PbCl2 kaolin gel Petiau 2 chamber type</Electrode>
      <Electrode location="S" number="40201037">Pb-PbCl2 kaolin gel Petiau 2 chamber type</Electrode>
    </Dipole>
    <Dipole name="Ey" type="wire">
      <Manufacturer>Oregon State University</Manufacturer>
      <Length units="meters">100.000</Length>
      <Azimuth units="degrees">99.063</Azimuth>
      <Electrode location="E" number="40201032">Pb-PbCl2 kaolin gel Petiau 2 chamber type</Electrode>
      <Electrode location="W" number="40201031">Pb-PbCl2 kaolin gel Petiau 2 chamber type</Electrode>
    </Dipole>
    <Comments author="Isaac Sageman">X array at 0 deg rotation. All e-lines 50m. Soft sandy dirt. Water tank ~400m NE. County Rd 601 ~200m SE. Warm sunny day.</Comments>
    <Errors></Errors>
    <SamplingRate units="Hz">1.000</SamplingRate>
    <Start>2020-09-20T19:03:06</Start>
    <End>2020-09-20T19:29:28</End>
  </FieldNotes>
  <FieldNotes run="NMX20b">
    <Instrument>
      <Manufacturer>Barry Narod</Manufacturer>
      <Name>NIMS</Name>
      <Id>2612-01</Id>
      <Settings/>
    </Instrument>
    <Magnetometer type="fluxgate">
      <Manufacturer>Barry Narod</Manufacturer>
      <Name>NIMS</Name>
      <Id>2509-23</Id>
      <Settings/>
    </Magnetometer>
    <Dipole name="Ex" type="wire">
      <Manufacturer>Oregon State University</Manufacturer>
      <Length units="meters">100.000</Length>
      <Azimuth units="degrees">9.117</Azimuth>
      <Electrode location="N" number="40201038">Pb-PbCl2 kaolin gel Petiau 2 chamber type</Electrode>
      <Electrode location="S" number="40201037">Pb-PbCl2 kaolin gel Petiau 2 chamber type</Electrode>
    </Dipole>
    <Dipole name="Ey" type="wire">
      <Manufacturer>Oregon State University</Manufacturer>
      <Length units="meters">100.000</Length>
      <Azimuth units="degrees">99.117</Azimuth>
      <Electrode location="E" number="40201032">Pb-PbCl2 kaolin gel Petiau 2 chamber type</Electrode>
      <Electrode location="W" number="40201031">Pb-PbCl2 kaolin gel Petiau 2 chamber type</Electrode>
    </Dipole>
    <Comments author="Isaac Sageman">X array at 0 deg rotation. All e-lines 50m. Soft sandy dirt. Water tank ~400m NE. County Rd 601 ~200m SE. Warm sunny day.</Comments>
    <Errors>Found data gaps (2). Gaps of unknown length: 1 [1469160].]</Errors>
    <SamplingRate units="Hz">1.000</SamplingRate>
    <Start>2020-09-20T20:12:29</Start>
    <End>2020-10-07T20:28:00</End>
  </FieldNotes>
  <ProcessingInfo>
    <SignConvention>exp(+ i\omega t)</SignConvention>
    <RemoteRef type="Robust Multi-Station Reference"/>
    <ProcessedBy>Jade Crosbie, Paul Bedrosian and Anna Kelbert</ProcessedBy>
    <ProcessingSoftware>
      <Name>EMTF</Name>
      <LastMod>2015-08-26</LastMod>
      <Author>Gary Egbert</Author>
    </ProcessingSoftware>
    <ProcessingTag>NMX20b_NMX20_NMW20_COR21_NMY21-NMX20b_NMX20_UTS18</ProcessingTag>
  </ProcessingInfo>
  <StatisticalEstimates>
    <Estimate name="VAR" type="real">
      <Description>Variance</Description>
      <ExternalUrl>http://www.iris.edu/dms/products/emtf/variance.html</ExternalUrl>
      <Intention>error estimate</Intention>
      <Tag>variance</Tag>
    </Estimate>
    <Estimate name="COV" type="complex">
      <Description>Full covariance between each two TF components</Description>
      <ExternalUrl>http://www.iris.edu/dms/products/emtf/covariance.html</ExternalUrl>
      <Intention>error estimate</Intention>
      <Tag>covariance</Tag>
    </Estimate>
    <Estimate name="INVSIGCOV" type="complex">
      <Description>Inverse Coherent Signal Power Matrix (S)</Description>
      <ExternalUrl>http://www.iris.edu/dms/products/emtf/inverse_signal_covariance.html</ExternalUrl>
      <Intention>signal power estimate</Intention>
      <Tag>inverse_signal_covariance</Tag>
    </Estimate>
    <Estimate name="RESIDCOV" type="complex">
      <Description>Residual Covariance (N)</Description>
      <ExternalUrl>http://www.iris.edu/dms/products/emtf/residual_covariance.html</ExternalUrl>
      <Intention>error estimate</Intention>
      <Tag>residual_covariance</Tag>
    </Estimate>
    <Estimate name="COH" type="complex">
      <Description>Coherence</Description>
      <ExternalUrl>http://www.iris.edu/dms/products/emtf/coherence.html</ExternalUrl>
      <Intention>signal coherence</Intention>
      <Tag>coherence</Tag>
    </Estimate>
    <Estimate name="PREDCOH" type="complex">
      <Description>Multiple Coherence</Description>
      <ExternalUrl>http://www.iris.edu/dms/products/emtf/multiple_coherence.html</ExternalUrl>
      <Intention>signal coherence</Intention>
      <Tag>multiple_coherence</Tag>
    </Estimate>
    <Estimate name="SIGAMP" type="complex">
      <Description>Signal Amplitude</Description>
      <ExternalUrl>http://www.iris.edu/dms/products/emtf/signal_amplitude.html</ExternalUrl>
      <Intention>signal power estimate</Intention>
      <Tag>signal_amplitude</Tag>
    </Estimate>
    <Estimate name="SIGNOISE" type="complex">
      <Description>Signal Noise</Description>
      <ExternalUrl>http://www.iris.edu/dms/products/emtf/signal_noise.html</ExternalUrl>
      <Intention>error estimate</Intention>
      <Tag>signal_noise</Tag>
    </Estimate>
  </StatisticalEstimates>
  <DataTypes>
    <DataType name="Z" type="complex" output="E" input="H" units="[mV/km]/[nT]">
      <Description>MT impedance</Description>
      <ExternalUrl>http://www.iris.edu/dms/products/emtf/impedance.html</ExternalUrl>
      <Intention>primary data type</Intention>
      <Tag>impedance</Tag>
    </DataType>
    <DataType name="T" type="complex" output="H" input="H" units="[]">
      <Description>Vertical Field Transfer Functions (Tipper)</Description>
      <ExternalUrl>http://www.iris.edu/dms/products/emtf/tipper.html</ExternalUrl>
      <Intention>primary data type</Intention>
      <Tag>tipper</Tag>
    </DataType>
  </DataTypes>
  <SiteLayout>
    <InputChannels ref="site" units="m">
      <Magnetic name="Hx" orientation="9.1" x="0.0" y="0.0" z="0.0"/>
      <Magnetic name="Hy" orientation="99.1" x="0.0" y="0.0" z="0.0"/>
    </InputChannels>
    <OutputChannels ref="site" units="m">
      <Magnetic name="Hz" orientation="9.1" x="0.0" y="0.0" z="0.0"/>
      <Electric name="Ex" orientation="9.1" x="-50.0" y="0.0" z="0.0" x2="50.0" y2="0.0" z2="0.0"/>
      <Electric name="Ey" orientation="99.1" x="0.0" y="-50.0" z="0.0" x2="0.0" y2="50.0" z2="0.0"/>
    </OutputChannels>
  </SiteLayout>
  <Data count="33">

Read EMTF XML

We can read this into an TF object or an EMTFXML object for comparison.

[3]:

tf_object = TF(fn=TF_XML)
tf_object.read()
xml_object = EMTFXML(fn=TF_XML)

EMTFXML structure

Each level of metadata in the file above is contained within its own object in mt_metadata for easy read/write. Here is a list of them plus a few more attributes.

[4]:

print(
    "\n\t".join(
        ["Metadata Objects:"]
        + [
            func
            for func in dir(xml_object)
            if not callable(getattr(xml_object, func)) and not func.startswith("_")
        ]
    )
)

Metadata Objects:
        attachment
        changed
        copyright
        data
        data_types
        description
        element_keys
        external_url
        field_notes
        fn
        logger
        notes
        period_range
        primary_data
        processing_info
        product_id
        provenance
        save_dir
        site
        site_layout
        station_metadata
        statistical_estimates
        sub_type
        survey_metadata
        tags

[5]:

xml_object.site

[5]:

{
    "site": {
        "acquired_by": "National Geoelectromagnetic Facility",
        "country": "USA",
        "data_quality_notes.comments.author": "Jade Crosbie, Paul Bedrosian and Anna Kelbert",
        "data_quality_notes.comments.value": "great TF from 10 to 10000 secs (or longer)",
        "data_quality_notes.good_from_period": 5.0,
        "data_quality_notes.good_to_period": 29127.0,
        "data_quality_notes.rating": 5,
        "end": "2020-10-07T20:28:00+00:00",
        "id": "NMX20",
        "location.datum": "WGS84",
        "location.elevation": 1940.05,
        "location.latitude": 34.470528,
        "location.longitude": -108.712288,
        "name": "Nations Draw, NM, USA",
        "orientation.angle_to_geographic_north": 0.0,
        "orientation.layout": "orthogonal",
        "project": "USMTArray",
        "run_list": "NMX20a NMX20b",
        "start": "2020-09-20T19:03:06+00:00",
        "survey": "CONUS South",
        "year_collected": 2020
    }
}

Translation to TF

The metadata is translated into the common metadata containers of the TF class

[6]:

tf_object.station_metadata

[6]:

{
    "station": {
        "acquired_by.author": "National Geoelectromagnetic Facility",
        "channels_recorded": [
            "ex",
            "ey",
            "hx",
            "hy",
            "hz"
        ],
        "comments": "description:Magnetotelluric Transfer Functions; primary_data.filename:NMX20b_NMX20_NMW20_COR21_NMY21-NMX20b_NMX20_UTS18.png; attachment.description:The original used to produce the XML; attachment.filename:NMX20b_NMX20_NMW20_COR21_NMY21-NMX20b_NMX20_UTS18.zmm; site.data_quality_notes.comments.author:Jade Crosbie, Paul Bedrosian and Anna Kelbert; site.data_quality_notes.comments.value:great TF from 10 to 10000 secs (or longer)",
        "data_type": "mt",
        "fdsn.id": "USMTArray.NMX20.2020",
        "geographic_name": "Nations Draw, NM, USA",
        "id": "NMX20",
        "location.datum": "WGS84",
        "location.declination.epoch": "2020.0",
        "location.declination.model": "WMM",
        "location.declination.value": 9.09,
        "location.elevation": 1940.05,
        "location.latitude": 34.470528,
        "location.longitude": -108.712288,
        "orientation.angle_to_geographic_north": 0.0,
        "orientation.method": null,
        "orientation.reference_frame": "geographic",
        "provenance.creation_time": "2021-03-17T14:47:44+00:00",
        "provenance.software.author": null,
        "provenance.software.name": "EMTF File Conversion Utilities 4.0",
        "provenance.software.version": null,
        "provenance.submitter.author": "Anna Kelbert",
        "provenance.submitter.email": "akelbert@usgs.gov",
        "provenance.submitter.organization": "U.S. Geological Survey, Geomagnetism Program",
        "release_license": "CC0-1.0",
        "run_list": [
            "NMX20a",
            "NMX20b"
        ],
        "time_period.end": "2020-10-07T20:28:00+00:00",
        "time_period.start": "2020-09-20T19:03:06+00:00",
        "transfer_function.coordinate_system": "geopgraphic",
        "transfer_function.id": "NMX20",
        "transfer_function.processed_date": null,
        "transfer_function.processing_parameters": [
            "{remote_ref.type: Robust Multi-Station Reference}"
        ],
        "transfer_function.remote_references": [
            "NMX20b",
            "NMX20",
            "NMW20",
            "COR21",
            "NMY21-NMX20b",
            "NMX20",
            "UTS18"
        ],
        "transfer_function.runs_processed": [
            "NMX20a",
            "NMX20b"
        ],
        "transfer_function.sign_convention": "exp(+ i\\omega t)",
        "transfer_function.units": null
    }
}

[7]:

tf_object.station_metadata.runs[0].ex

[7]:

{
    "electric": {
        "channel_number": 0,
        "component": "ex",
        "data_quality.rating.value": 0,
        "dipole_length": 100.0,
        "filter.applied": [
            false
        ],
        "filter.name": [],
        "measurement_azimuth": 9.1,
        "measurement_tilt": 0.0,
        "negative.elevation": 0.0,
        "negative.id": "40201037",
        "negative.latitude": 0.0,
        "negative.longitude": 0.0,
        "negative.manufacturer": "Oregon State University",
        "negative.type": "Pb-PbCl2 kaolin gel Petiau 2 chamber type",
        "negative.x": -50.0,
        "negative.y": 0.0,
        "negative.z": 0.0,
        "positive.elevation": 0.0,
        "positive.id": "40201038",
        "positive.latitude": 0.0,
        "positive.longitude": 0.0,
        "positive.manufacturer": "Oregon State University",
        "positive.type": "Pb-PbCl2 kaolin gel Petiau 2 chamber type",
        "positive.x2": 50.0,
        "positive.y2": 0.0,
        "positive.z2": 0.0,
        "sample_rate": 0.0,
        "time_period.end": "1980-01-01T00:00:00+00:00",
        "time_period.start": "1980-01-01T00:00:00+00:00",
        "translated_azimuth": 9.1,
        "type": "electric",
        "units": null
    }
}

EMTF XML

EMTF XML file

Read EMTF XML

EMTFXML structure

Translation to TF

EMTF XML Statistical Estimates