mt_metadata.base package

Submodules

mt_metadata.base.helpers module

Created on Wed Dec 23 20:37:52 2020

copyright:: Jared Peacock (jpeacock@usgs.gov)
license:: MIT

class mt_metadata.base.helpers.NumpyEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]

Bases: JSONEncoder

Need to encode numpy ints and floats for json to work

default(obj)[source]

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)

mt_metadata.base.helpers.dict_to_xml(meta_dict, attr_dict=None)[source]

Assumes dictionary is structured {class:{attribute_dict}}

Parameters:: meta_dict (TYPE) – DESCRIPTION
Returns:: DESCRIPTION
Return type:: TYPE

mt_metadata.base.helpers.element_to_dict(element)[source]

Parameters:: element (TYPE) – DESCRIPTION
Returns:: DESCRIPTION
Return type:: TYPE

mt_metadata.base.helpers.element_to_string(element)[source]

mt_metadata.base.helpers.flatten_dict(meta_dict, parent_key=None, sep='.')[source]

Parameters:

meta_dict (TYPE) – DESCRIPTION
parent_key (TYPE, optional) – DESCRIPTION, defaults to None
sep (TYPE, optional) – DESCRIPTION, defaults to ‘.’

Returns:

DESCRIPTION

Return type:

TYPE

mt_metadata.base.helpers.flatten_list(x_list)[source]

Flatten a nested list flatten = lambda l: [item for sublist in l for item in sublist]

Return type:: None.

mt_metadata.base.helpers.get_type(name, attr_dict)[source]

mt_metadata.base.helpers.get_units(name, attr_dict)[source]

mt_metadata.base.helpers.recursive_split_dict(key, value, remainder, sep='.')[source]

recursively split a dictionary

Parameters:

key (TYPE) – DESCRIPTION
value (TYPE) – DESCRIPTION
remainder (TYPE) – DESCRIPTION

Returns:

DESCRIPTION

Return type:

TYPE

mt_metadata.base.helpers.recursive_split_getattr(base_object, name, sep='.')[source]

mt_metadata.base.helpers.recursive_split_setattr(base_object, name, value, sep='.')[source]

mt_metadata.base.helpers.recursive_split_xml(element, item, base, name, attr_dict=None)[source]

mt_metadata.base.helpers.structure_dict(meta_dict, sep='.')[source]

Parameters:

meta_dict (TYPE) – DESCRIPTION
sep (TYPE, optional) – DESCRIPTION, defaults to ‘.’

Returns:

DESCRIPTION

Return type:

TYPE

mt_metadata.base.helpers.validate_c1(attr_dict, c1)[source]

Parameters:

attr_dict (TYPE) – DESCRIPTION
c1 (TYPE) – DESCRIPTION

Returns:

DESCRIPTION

Return type:

TYPE

mt_metadata.base.helpers.validate_name(name, pattern=None)[source]

Validate name

Parameters:

name (TYPE) – DESCRIPTION
pattern (TYPE, optional) – DESCRIPTION, defaults to None

Returns:

DESCRIPTION

Return type:

TYPE

mt_metadata.base.helpers.wrap_description(description, column_width)[source]: split a description into separate lines

mt_metadata.base.helpers.write_block(key, attr_dict, c1=45, c2=45, c3=15)[source]

Parameters:

key (string) – key to write from attr dict
attr_dict (dict) – attribute dictionary
c1 (int, optional) – column 1 width, defaults to 45
c2 (int, optional) – column 2 width, defaults to 45
c3 (int, optional) – column 3 width, defaults to 15

Returns:

list of lines

Return type:

list

mt_metadata.base.helpers.write_lines(attr_dict, c1=45, c2=45, c3=15)[source]

Takes the attribute dictionary from the json and parses it into a table Returns a string representation of this table. This overwrites the doc.

Parameters:

attr_dict (dict) – attribute dictionary
c1 (integer, optional) – cloumn 1 width, defaults to 45
c2 (integer, optional) – column 2 width, defaults to 45
c3 (integer, optional) – column 3 width, defaults to 15

Returns:

doc string

Return type:

string

mt_metadata.base.metadata module

Created on Wed Dec 23 20:41:16 2020

copyright:: Jared Peacock (jpeacock@usgs.gov)
license:: MIT

class mt_metadata.base.metadata.Base(attr_dict={}, **kwargs)[source]

Bases: object

add_base_attribute(name, value, value_dict)[source]

Add an attribute to _attr_dict so it will be included in the output dictionary

Parameters:

name (string) – name of attribute
value (described in value_dict) – value of the new attribute
value_dict –
dictionary describing the attribute, must have keys [‘type’, ‘required’, ‘style’, ‘units’, ‘alias’, ‘description’,

’options’, ‘example’]

type –> the data type [ str | int | float | bool ]
required –> required in the standards [ True | False ]
style –> style of the string
units –> units of the attribute, must be a string
alias –> other possible names for the attribute
options –> if only a few options are accepted, separated by | or
comma.b [ option_01 | option_02 | other ]. ‘other’ means other options available but not yet defined.
example –> an example of the attribute

Example:

>>> extra = {'type': str,
>>> ...      'style': 'controlled vocabulary',
>>> ...      'required': False,
>>> ...      'units': celsius,
>>> ...      'description': 'local temperature',
>>> ...      'alias': ['temp'],
>>> ...      'options': [ 'ambient', 'air', 'other'],
>>> ...      'example': 'ambient'}
>>> r = Run()
>>> r.add_base_attribute('temperature', 'ambient', extra)

attribute_information(name=None)[source]

return a descriptive string of the attribute if none returns for all

Parameters:: name (string, optional) – attribute name for a specifice attribute, defaults to None
Returns:: description of the attributes or specific attribute if asked
Return type:: string

property changed

copy()[source]: Copy object

from_dict(meta_dict, skip_none=False)[source]

fill attributes from a dictionary

Parameters:: meta_dict (dictionary) – dictionary with keys equal to metadata.

from_json(json_str)[source]

read in a json string and update attributes of an object

Parameters:: json_str (string or pathlib.Path) – json string or file path

from_series(pd_series)[source]

Fill attributes from a Pandas series

Note

Currently, the series must be single layered with key names separated by dots. (location.latitude)

Parameters:: pd_series (pandas.Series) – Series containing metadata information

from_xml(xml_element)[source]

Parameters:: xml_element (etree.Element) – XML element
Returns:: Fills attributes accordingly

get_attr_from_name(name)[source]

Access attribute from the given name.

The name can contain the name of an object which must be separated by a ‘.’ for e.g. {object_name}.{name} –> location.latitude

Note

this is a helper function for names with ‘.’ in the name for easier getting when reading from dictionary.

Parameters:: name (string) – name of attribute to get.
Returns:: attribute value
Return type:: type is defined by the attribute name
Example:

>>> b = Base(**{'category.test_attr':10})
>>> b.get_attr_from_name('category.test_attr')
10

get_attribute_list()[source]: return a list of the attributes

set_attr_from_name(name, value)[source]

Helper function to set attribute from the given name.

The name can contain the name of an object which must be separated by a ‘.’ for e.g. {object_name}.{name} –> location.latitude

Note

this is a helper function for names with ‘.’ in the name for easier getting when reading from dictionary.

Parameters:

name (string) – name of attribute to get.
value (type is defined by the attribute name) – attribute value

Example:

>>> b = Base(**{'category.test_attr':10})
>>> b.set_attr_from_name('category.test_attr', '10')
>>> print(b.category.test_attr)
'10'

to_dict(nested=False, single=False, required=True)[source]

make a dictionary from attributes, makes dictionary from _attr_list.

Parameters:

nested ([ True | False ] , default is False) – make the returned dictionary nested
single ([ True | False ], default is False) – return just metadata dictionary -> meta_dict[class_name]
required – return just the required elements and any elements with non-None values

to_json(nested=False, indent=' ', required=True)[source]

Write a json string from a given object, taking into account other class objects contained within the given object.

Parameters:: nested ([ True | False ] , default is False) – make the returned json nested

to_series(required=True)[source]

Convert attribute list to a pandas.Series

Note

this is a flattened version of the metadata

Returns:: pandas.Series
Return type:: pandas.Series

to_xml(string=False, required=True)[source]

make an xml element for the attribute that will add types and units.

Parameters:: string ([ True | False ], default is False) – output a string instead of an XML element
Returns:: XML element or string

update(other, match=[])[source]

Update attribute values from another like element, skipping None

Parameters:: other (mt_metadata.base.metadata.Base) – other Base object

mt_metadata.base.schema module

Created on Thu Dec 24 12:02:12 2020

copyright:: Jared Peacock (jpeacock@usgs.gov)
license:: MIT

class mt_metadata.base.schema.BaseDict(*args, **kwargs)[source]

Bases: MutableMapping

BaseDict is a convenience class that can help the metadata dictionaries act like classes so you can access variables by .name or [name]

Note

If the attribute has a . in the name then you will not be able to access that attribute by class.name.name You will get an attribute error. You need to access the attribute like a dictionary class[‘name.name’]

You can add an attribute by:

>>> b = BaseDict()
>>> b.update({name: value_dict})

Or you can add a whole dictionary:

>>> b.add_dict(ATTR_DICT['run'])

All attributes have a descriptive dictionary of the form:

>>> {'type': data type, 'required': [True | False],
>>> ... 'style': 'string style', 'units': attribute units}
type –> the data type [ str | int | float | bool ]

required –> required in the standards [ True | False ]

style –> style of the string

units –> units of the attribute, must be a string

add_dict(add_dict, name=None, keys=None)[source]

Add a dictionary to. If name is input it is added to the keys of the input dictionary

Parameters:

add_dict (dictionary, or MutableMapping) – dictionary to add
name (string or None) – name to add to keys

Example:

>>> s_obj = Standards()
>>> run_dict = s_obj.run_dict
>>> run_dict.add_dict(s_obj.declination_dict, 'declination')

copy()[source]

from_csv(csv_fn)[source]

Read in CSV file as a dictionary

Parameters:

csv_fn (pathlib.Path or string) – csv file to read metadata standards from

Returns:

dictionary of the contents of the file

Return type:

Dictionary

Example:

>>> run_dict = BaseDict()
>>> run_dict.from_csv(get_level_fn('run'))

from_json(json_fn)[source]

Read schema standards from json

Parameters:: json_fn (string or Path) – full path to json file
Returns:: full path to json file
Return type:: Path

property name

to_csv(csv_fn)[source]

write dictionary to csv file

Parameters:

level_dict (TYPE) – DESCRIPTION
csv_fn (TYPE) – DESCRIPTION

Returns:

DESCRIPTION

Return type:

TYPE

to_json(json_fn, indent=' ')[source]

Write schema standards to json

Parameters:: json_fn (string or Path) – full path to json file
Returns:: full path to json file
Return type:: Path

to_latex(max_entries=7, first_table_len=7)[source]

Parameters:: level_dict (TYPE) – DESCRIPTION
Returns:: DESCRIPTION
Return type:: TYPE

mt_metadata.base.schema.get_schema(schema_element, paths)[source]

Get a mt_metadata.schema_base.BaseDict object of the element

Parameters:: schema_element (string) – name of the schema element to get filename for
Returns:: return a dictionary that describes the standards for the element
Return type:: mt_metadata.schema_base.BaseDict

mt_metadata.base.schema.get_schema_fn(schema_element, paths)[source]

Get the correct file name for the given schema element from the provided list of valid file names

Parameters:: schema_element (string) – name of the schema element to get filename for
Returns:: correct file name for given element
Return type:: pathlib.Path

Module contents

Base classes for holding metadata and schema objects

class mt_metadata.base.Base(attr_dict={}, **kwargs)[source]

Bases: object

add_base_attribute(name, value, value_dict)[source]

Add an attribute to _attr_dict so it will be included in the output dictionary

Parameters:

name (string) – name of attribute
value (described in value_dict) – value of the new attribute
value_dict –
dictionary describing the attribute, must have keys [‘type’, ‘required’, ‘style’, ‘units’, ‘alias’, ‘description’,

’options’, ‘example’]

type –> the data type [ str | int | float | bool ]
required –> required in the standards [ True | False ]
style –> style of the string
units –> units of the attribute, must be a string
alias –> other possible names for the attribute
options –> if only a few options are accepted, separated by | or
comma.b [ option_01 | option_02 | other ]. ‘other’ means other options available but not yet defined.
example –> an example of the attribute

Example:

>>> extra = {'type': str,
>>> ...      'style': 'controlled vocabulary',
>>> ...      'required': False,
>>> ...      'units': celsius,
>>> ...      'description': 'local temperature',
>>> ...      'alias': ['temp'],
>>> ...      'options': [ 'ambient', 'air', 'other'],
>>> ...      'example': 'ambient'}
>>> r = Run()
>>> r.add_base_attribute('temperature', 'ambient', extra)

attribute_information(name=None)[source]

return a descriptive string of the attribute if none returns for all

Parameters:: name (string, optional) – attribute name for a specifice attribute, defaults to None
Returns:: description of the attributes or specific attribute if asked
Return type:: string

property changed

copy()[source]: Copy object

from_dict(meta_dict, skip_none=False)[source]

fill attributes from a dictionary

Parameters:: meta_dict (dictionary) – dictionary with keys equal to metadata.

from_json(json_str)[source]

read in a json string and update attributes of an object

Parameters:: json_str (string or pathlib.Path) – json string or file path

from_series(pd_series)[source]

Fill attributes from a Pandas series

Note

Currently, the series must be single layered with key names separated by dots. (location.latitude)

Parameters:: pd_series (pandas.Series) – Series containing metadata information

from_xml(xml_element)[source]

Parameters:: xml_element (etree.Element) – XML element
Returns:: Fills attributes accordingly

get_attr_from_name(name)[source]

Access attribute from the given name.

The name can contain the name of an object which must be separated by a ‘.’ for e.g. {object_name}.{name} –> location.latitude

Note

this is a helper function for names with ‘.’ in the name for easier getting when reading from dictionary.

Parameters:: name (string) – name of attribute to get.
Returns:: attribute value
Return type:: type is defined by the attribute name
Example:

>>> b = Base(**{'category.test_attr':10})
>>> b.get_attr_from_name('category.test_attr')
10

get_attribute_list()[source]: return a list of the attributes

set_attr_from_name(name, value)[source]

Helper function to set attribute from the given name.

The name can contain the name of an object which must be separated by a ‘.’ for e.g. {object_name}.{name} –> location.latitude

Note

this is a helper function for names with ‘.’ in the name for easier getting when reading from dictionary.

Parameters:

name (string) – name of attribute to get.
value (type is defined by the attribute name) – attribute value

Example:

>>> b = Base(**{'category.test_attr':10})
>>> b.set_attr_from_name('category.test_attr', '10')
>>> print(b.category.test_attr)
'10'

to_dict(nested=False, single=False, required=True)[source]

make a dictionary from attributes, makes dictionary from _attr_list.

Parameters:

nested ([ True | False ] , default is False) – make the returned dictionary nested
single ([ True | False ], default is False) – return just metadata dictionary -> meta_dict[class_name]
required – return just the required elements and any elements with non-None values

to_json(nested=False, indent=' ', required=True)[source]

Write a json string from a given object, taking into account other class objects contained within the given object.

Parameters:: nested ([ True | False ] , default is False) – make the returned json nested

to_series(required=True)[source]

Convert attribute list to a pandas.Series

Note

this is a flattened version of the metadata

Returns:: pandas.Series
Return type:: pandas.Series

to_xml(string=False, required=True)[source]

make an xml element for the attribute that will add types and units.

Parameters:: string ([ True | False ], default is False) – output a string instead of an XML element
Returns:: XML element or string

update(other, match=[])[source]

Update attribute values from another like element, skipping None

Parameters:: other (mt_metadata.base.metadata.Base) – other Base object

class mt_metadata.base.BaseDict(*args, **kwargs)[source]

Bases: MutableMapping

BaseDict is a convenience class that can help the metadata dictionaries act like classes so you can access variables by .name or [name]

Note

If the attribute has a . in the name then you will not be able to access that attribute by class.name.name You will get an attribute error. You need to access the attribute like a dictionary class[‘name.name’]

You can add an attribute by:

>>> b = BaseDict()
>>> b.update({name: value_dict})

Or you can add a whole dictionary:

>>> b.add_dict(ATTR_DICT['run'])

All attributes have a descriptive dictionary of the form:

>>> {'type': data type, 'required': [True | False],
>>> ... 'style': 'string style', 'units': attribute units}
type –> the data type [ str | int | float | bool ]

required –> required in the standards [ True | False ]

style –> style of the string

units –> units of the attribute, must be a string

add_dict(add_dict, name=None, keys=None)[source]

Add a dictionary to. If name is input it is added to the keys of the input dictionary

Parameters:

add_dict (dictionary, or MutableMapping) – dictionary to add
name (string or None) – name to add to keys

Example:

>>> s_obj = Standards()
>>> run_dict = s_obj.run_dict
>>> run_dict.add_dict(s_obj.declination_dict, 'declination')

copy()[source]

from_csv(csv_fn)[source]

Read in CSV file as a dictionary

Parameters:

csv_fn (pathlib.Path or string) – csv file to read metadata standards from

Returns:

dictionary of the contents of the file

Return type:

Dictionary

Example:

>>> run_dict = BaseDict()
>>> run_dict.from_csv(get_level_fn('run'))

from_json(json_fn)[source]

Read schema standards from json

Parameters:: json_fn (string or Path) – full path to json file
Returns:: full path to json file
Return type:: Path

property name

to_csv(csv_fn)[source]

write dictionary to csv file

Parameters:

level_dict (TYPE) – DESCRIPTION
csv_fn (TYPE) – DESCRIPTION

Returns:

DESCRIPTION

Return type:

TYPE

to_json(json_fn, indent=' ')[source]

Write schema standards to json

Parameters:: json_fn (string or Path) – full path to json file
Returns:: full path to json file
Return type:: Path

to_latex(max_entries=7, first_table_len=7)[source]

Parameters:: level_dict (TYPE) – DESCRIPTION
Returns:: DESCRIPTION
Return type:: TYPE

mt_metadata.base.get_schema(schema_element, paths)[source]

Get a mt_metadata.schema_base.BaseDict object of the element

Parameters:: schema_element (string) – name of the schema element to get filename for
Returns:: return a dictionary that describes the standards for the element
Return type:: mt_metadata.schema_base.BaseDict