utilities

Download and management utilities for syncing time and auxiliary files

Can list a directory on a ftp host

Can download a file from a ftp or http host

Can download a file from CDDIS via https when NASA Earthdata credentials are supplied

Checks MD5 or sha1 hashes between local and remote files

Source code

General Methods

timescale.utilities.get_data_path(relpath: list | str | Path)[source]

Get the absolute path within a package from a relative path

Parameters:

relpath: list, str or pathlib.Path: Relative path

class timescale.utilities.reify(wrapped)[source]: Class decorator that puts the result of the method it decorates into the instance

timescale.utilities.file_opener(filename: str | Path)[source]

Platform independent file opener

Parameters:

filename: str or pathlib.Path: path to file

timescale.utilities.get_hash(local: str | IOBase | Path, algorithm: str = 'md5')[source]

Get the hash value from a local file or BytesIO object

Parameters:

local: obj, str or pathlib.Path: BytesIO object or path to file
algorithm: str, default ‘md5’: Hashing algorithm for checksum validation

timescale.utilities.get_git_revision_hash(refname: str = 'HEAD', short: bool = False)[source]

Get the git hash value for a particular reference

Parameters:

refname: str, default HEAD: Symbolic reference name
short: bool, default False: Return the shorted hash value

timescale.utilities.get_git_status()[source]: Get the status of a git repository as a boolean value

timescale.utilities.url_split(s: str)[source]

Recursively split a URL path into a list

Parameters:

s: str: URL string

timescale.utilities.roman_to_int(roman: str)[source]

Converts a string from Roman numerals into an integer (Arabic)

Parameters:

roman: str: Roman numeral string

timescale.utilities.get_unix_time(time_string: str, format: str = '%Y-%m-%d %H:%M:%S')[source]

Get the Unix timestamp value for a formatted date string

Parameters:

time_string: str: Formatted time string to parse
format: str, default ‘%Y-%m-%d %H:%M:%S’: Format for input time string

timescale.utilities.isoformat(time_string: str)[source]

Reformat a date string to ISO formatting

Parameters:

time_string: str: formatted time string to parse

timescale.utilities.even(value: float)[source]

Rounds a number to an even number less than or equal to original

Parameters:

value: float: Number to be rounded

timescale.utilities.ceil(value: float)[source]

Rounds a number upward to its nearest integer

Parameters:

value: float: Number to be rounded upward

timescale.utilities.copy(source: str | Path, destination: str | Path, move: bool = False, **kwargs)[source]

Copy or move a file with all system information

Parameters:

source: str: Source file
destination: str: Copied destination file
move: bool, default False: Remove the source file

timescale.utilities.check_ftp_connection(HOST: str, username: str | None = None, password: str | None = None, encrypted: bool = False)[source]

Check internet connection with ftp host

Parameters:

HOST: str: Remote ftp host
username: str or NoneType: ftp username
password: str or NoneType: ftp password
encrypted: bool, default False: Use an encrypted (TLS) connection

List a directory on a ftp host

Parameters:

HOST: str or list: Remote ftp host path split as list
username: str or NoneType: ftp username
password: str or NoneType: ftp password
encrypted: bool, default False: Use an encrypted (TLS) connection
timeout: int or NoneType, default None: Timeout in seconds for blocking operations
basename: bool, default False: Return the file or directory basename instead of the full path
pattern: str or NoneType, default None: Regular expression pattern for reducing list
sort: bool, default False: Sort output list

Returns:

output: list: Items in a directory
mtimes: list: Last modification times for items in the directory

timescale.utilities.from_ftp(HOST: str | list, username: str | None = None, password: str | None = None, timeout: int | None = None, encrypted: bool = False, local: str | ~pathlib.Path | None = None, hash: str = '', chunk: int = 8192, verbose: bool = False, fid: object = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, label: str | None = None, mode: oct = 509, **kwargs)[source]

Download a file from a ftp host

Parameters:

HOST: str or list: Remote ftp host path
username: str or NoneType: ftp username
password: str or NoneType: ftp password
encrypted: bool, default False: Use an encrypted (TLS) connection
timeout: int or NoneType, default None: Timeout in seconds for blocking operations
local: str, pathlib.Path or NoneType, default None: Path to local file
hash: str, default ‘’: MD5 hash of local file
chunk: int, default 8192: Chunk size for transfer encoding
verbose: bool, default False: Print file transfer information
fid: object, default sys.stdout: Open file object for logging file transfers if verbose
label: str, default None: Label for logging file transfer information if verbose
mode: oct, default 0o775: Permissions mode of output local file

Returns:

remote_buffer: obj: BytesIO representation of file

timescale.utilities._create_default_ssl_context() → SSLContext[source]: Creates the default SSL context

timescale.utilities._create_ssl_context_no_verify() → SSLContext[source]: Creates an SSL context for unverified connections

timescale.utilities._set_ssl_context_options(context: SSLContext) → None[source]: Sets the default options for the SSL context

timescale.utilities.check_connection(HOST: str, context: ~ssl.SSLContext = <ssl.SSLContext object>, timeout: int = 20)[source]

Check internet connection with http host

Parameters:

HOST: str: Remote http host
context: obj, default timescale.utilities._default_ssl_context: SSL context for urllib opener object
timeout: int, default 20: Timeout in seconds for blocking operations

timescale.utilities.http_list(HOST: str | list, timeout: int | None = None, context: ~ssl.SSLContext = <ssl.SSLContext object>, parser=<lxml.etree.HTMLParser object>, format: str = '%Y-%m-%d %H:%M', pattern: str = '', sort: bool = False, **kwargs)[source]

List a directory on an Apache http Server

Parameters:

HOST: str or list: Remote http host path
timeout: int or NoneType, default None: Timeout in seconds for blocking operations
context: obj, default timescale.utilities._default_ssl_context: SSL context for urllib opener object
parser: obj, default lxml.etree.HTMLParser(): HTML parser for lxml
format: str, default ‘%Y-%m-%d %H:%M’: Format for input time string
pattern: str, default ‘’: Regular expression pattern for reducing list
sort: bool, default False: Sort output list

Returns:

colnames: list: Column names in a directory
collastmod: list: Last modification times for items in the directory

timescale.utilities.from_http(HOST: str | list, timeout: int | None = None, context: ~ssl.SSLContext = <ssl.SSLContext object>, local: str | ~pathlib.Path | None = None, hash: str = '', chunk: int = 16384, headers: dict = {}, verbose: bool = False, fid: object = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, label: str | None = None, mode: oct = 509, **kwargs)[source]

Download a file from a http host

Parameters:

HOST: str or list: Remote http host path split as list
timeout: int or NoneType, default None: Timeout in seconds for blocking operations
context: obj, default timescale.utilities._default_ssl_context: SSL context for urllib opener object
local: str, pathlib.Path or NoneType, default None: Path to local file
hash: str, default ‘’: MD5 hash of local file
chunk: int, default 16384: Chunk size for transfer encoding
headers: dict, default {}: Dictionary of headers to append from URL request
verbose: bool, default False: Print file transfer information
fid: object, default sys.stdout: Open file object for logging file transfers if verbose
label: str or None, default None: Label for logging file transfer information if verbose
mode: oct, default 0o775: Permissions mode of output local file

Returns:

remote_buffer: obj: BytesIO representation of file

timescale.utilities.attempt_login(urs: str, context: ~ssl.SSLContext = <ssl.SSLContext object>, password_manager: bool = True, get_ca_certs: bool = True, redirect: bool = True, authorization_header: bool = False, **kwargs)[source]

Attempt to build a urllib opener for NASA Earthdata

Parameters:

urs: str: Earthdata login URS 3 host
context: obj, default timescale.utilities._default_ssl_context: SSL context for urllib opener object
password_manager: bool, default True: Create password manager context using default realm
get_ca_certs: bool, default True: Get list of loaded “certification authority” certificates
redirect: bool, default True: Create redirect handler object
authorization_header: bool, default False: Add base64 encoded authorization header to opener
username: str, default from environmental variable: NASA Earthdata username
password: str, default from environmental variable: NASA Earthdata password
endpoint: str, default ‘https://cddis.nasa.gov/archive’: Full URL to a protected credential website
retries: int, default 5: number of retry attempts
netrc: str, default ~/.netrc: path to .netrc file for authentication

Returns:

opener: obj: OpenerDirector instance

timescale.utilities.build_opener(username: str, password: str, context: ~ssl.SSLContext = <ssl.SSLContext object>, password_manager: bool = True, get_ca_certs: bool = True, redirect: bool = True, authorization_header: bool = False, urs: str = 'https://urs.earthdata.nasa.gov')[source]

Build urllib opener for NASA Earthdata with supplied credentials

Parameters:

username: str or NoneType, default None: NASA Earthdata username
password: str or NoneType, default None: NASA Earthdata password
context: obj, default timescale.utilities._default_ssl_context: SSL context for urllib opener object
password_manager: bool, default True: Create password manager context using default realm
get_ca_certs: bool, default True: Get list of loaded “certification authority” certificates
redirect: bool, default True: Create redirect handler object
authorization_header: bool, default False: Add base64 encoded authorization header to opener
urs: str, default ‘https://urs.earthdata.nasa.gov’: Earthdata login URS 3 host

Returns:

opener: object: OpenerDirector instance

timescale.utilities.get_token(HOST: str = 'https://urs.earthdata.nasa.gov/api/users/token', username: str | None = None, password: str | None = None, build: bool = True, context: ~ssl.SSLContext = <ssl.SSLContext object>, urs: str = 'urs.earthdata.nasa.gov')[source]

Generate a NASA Earthdata User Token

Parameters:

HOST: str or list: NASA Earthdata token API host
username: str or NoneType, default None: NASA Earthdata username
password: str or NoneType, default None: NASA Earthdata password
build: bool, default True: Build opener and check credentials
timeout: int or NoneType, default None: timeout in seconds for blocking operations
context: obj, default timescale.utilities._default_ssl_context: SSL context for urllib opener object
urs: str, default ‘urs.earthdata.nasa.gov’: NASA Earthdata URS 3 host

Returns:

token: dict: JSON response with NASA Earthdata User Token

timescale.utilities.list_tokens(HOST: str = 'https://urs.earthdata.nasa.gov/api/users/tokens', username: str | None = None, password: str | None = None, build: bool = True, context: ~ssl.SSLContext = <ssl.SSLContext object>, urs: str = 'urs.earthdata.nasa.gov')[source]

List the current associated NASA Earthdata User Tokens

Parameters:

HOST: str: NASA Earthdata list token API host
username: str or NoneType, default None: NASA Earthdata username
password: str or NoneType, default None: NASA Earthdata password
build: bool, default True: Build opener and check credentials
timeout: int or NoneType, default None: timeout in seconds for blocking operations
context: obj, default timescale.utilities._default_ssl_context: SSL context for urllib opener object
urs: str, default ‘urs.earthdata.nasa.gov’: NASA Earthdata URS 3 host

Returns:

tokens: list: JSON response with NASA Earthdata User Tokens

timescale.utilities.revoke_token(token: str, HOST: str = 'https://urs.earthdata.nasa.gov/api/users/revoke_token', username: str | None = None, password: str | None = None, build: bool = True, context: ~ssl.SSLContext = <ssl.SSLContext object>, urs: str = 'urs.earthdata.nasa.gov')[source]

Generate a NASA Earthdata User Token

Parameters:

token: str: NASA Earthdata token to be revoked
HOST: str: NASA Earthdata revoke token API host
username: str or NoneType, default None: NASA Earthdata username
password: str or NoneType, default None: NASA Earthdata password
build: bool, default True: Build opener and check credentials
timeout: int or NoneType, default None: timeout in seconds for blocking operations
context: obj, default timescale.utilities._default_ssl_context: SSL context for urllib opener object
urs: str, default ‘urs.earthdata.nasa.gov’: NASA Earthdata URS 3 host

timescale.utilities.check_credentials(HOST: str = 'https://cddis.nasa.gov/archive')[source]

Check that entered NASA Earthdata credentials are valid

Parameters:

HOST: str, default ‘https://cddis.nasa.gov/archive’: Full URL to a protected credential website

timescale.utilities.cddis_list(HOST: str | list, username: str | None = None, password: str | None = None, build: bool = True, timeout: int | None = None, urs: str = 'urs.earthdata.nasa.gov', parser=<lxml.etree.HTMLParser object>, pattern: str = '', sort: bool = False)[source]

List a directory on GSFC CDDIS archive server

Parameters:

HOST: str or list: remote https host
username: str or NoneType, default None: NASA Earthdata username
password: str or NoneType, default None: NASA Earthdata password
build: bool, default True: Build opener and check Earthdata credentials
timeout: int or NoneType, default None: timeout in seconds for blocking operations
urs: str, default ‘urs.earthdata.nasa.gov’: NASA Earthdata URS 3 host
parser: obj, default lxml.etree.HTMLParser(): HTML parser for lxml
pattern: str, default ‘’: regular expression pattern for reducing list
sort: bool, default False: sort output list

Returns:

colnames: list: column names in a directory
collastmod: list: last modification times for items in the directory

timescale.utilities.from_cddis(HOST: str | list, username: str | None = None, password: str | None = None, build: bool = True, timeout: int | None = None, urs: str = 'urs.earthdata.nasa.gov', local: str | ~pathlib.Path | None = None, hash: str = '', chunk: int = 16384, verbose: bool = False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, mode: oct = 509)[source]

Download a file from GSFC CDDIS archive server

Parameters:

HOST: str or list: remote https host
username: str or NoneType, default None: NASA Earthdata username
password: str or NoneType, default None: NASA Earthdata password
build: bool, default True: Build opener and check Earthdata credentials
timeout: int or NoneType, default None: timeout in seconds for blocking operations
urs: str, default ‘urs.earthdata.nasa.gov’: NASA Earthdata URS 3 host
local: str, pathlib.Path or NoneType, default None: path to local file
hash: str, default ‘’: MD5 hash of local file
chunk: int, default 16384: chunk size for transfer encoding
verbose: bool, default False: print file transfer information
fid: obj, default sys.stdout: open file object to print if verbose
mode: oct, default 0o775: permissions mode of output local file

Returns:

remote_buffer: obj: BytesIO representation of file

timescale.utilities.iers_list(HOST: str | list, timeout: int | None = None, context: ~ssl.SSLContext = <ssl.SSLContext object>, parser=<lxml.etree.HTMLParser object>)[source]

List a directory on IERS Bulletin-A https server

Parameters:

HOST: str or list: Remote http host path
timeout: int or NoneType, default None: Timeout in seconds for blocking operations
context: obj, default timescale.utilities._default_ssl_context: SSL context for urllib opener object
parser: obj, default lxml.etree.HTMLParser(): HTML parser for lxml

Returns:

colnames: list: Column names in a directory
collastmod: list: Last modification times for items in the directory

timescale.utilities.from_jpl_ssd(kernel='de440s.bsp', timeout: int | None = None, context: ~ssl.SSLContext = <ssl.SSLContext object>, local: str | ~pathlib.Path | None = None, hash: str = '', chunk: int = 16384, verbose: bool = False, mode: oct = 509)[source]

Download planetary ephemeride kernels from the JPL Solar System Dynamics server

Parameters:

kernel: str: JPL kernel file to download
timeout: int or NoneType, default None: Timeout in seconds for blocking operations
context: obj, default timescale.utilities._default_ssl_context: SSL context for urllib opener object
hash: str, default ‘’: MD5 hash of local file
chunk: int, default 16384: Chunk size for transfer encoding
verbose: bool, default False: Print file transfer information
mode: oct, default 0o775: Permissions mode of output local file