pylablib.core.utils package¶

Submodules¶

pylablib.core.utils.array_utils module¶

pylablib.core.utils.array_utils.as_array(data, force_copy=False, try_object=True)[source]¶

Turn data into a numpy array.

If force_copy==True, copy the data if it’s already a numpy array. If try_object==False, only try to convert to numerical numpy arrays; otherwise, generic numpy arrays (with dtype=="object") are acceptable.

pylablib.core.utils.array_utils.get_shape(data, strict=False)[source]¶

Get the data shape.

If the data is a nested list and strict==True, raise an error unless all sublists have the same length (i.e., the data is rectangular).

pylablib.core.utils.ctypes_wrap module¶

pylablib.core.utils.ctypes_wrap.get_value(rval)[source]¶: Get value of a ctypes variable

pylablib.core.utils.ctypes_wrap.setup_func(func, argtypes, restype=None, errcheck=None)[source]¶

Setup a ctypes function.

Assign argtypes (list of argument types), restype (return value type) and errcheck (error checking function called for the return value).

class pylablib.core.utils.ctypes_wrap.CFunctionWrapper(restype=None, errcheck=None, tuple_single_retval=False, return_res='auto', default_rvals='rest', pointer_byref=False)[source]¶

Bases: object

Wrapper object for ctypes function.

The main methods are wrap_annotated() and wrap_bare(), which wrap a ctypes function and returns a Python function with a proper signature. These methods can also handle some standard use cases such as passing parameters by reference, or setting up the function arguments, or parsing the results. These methods can also be invoked when the wrapper is used as a callable; in this case, the exact method is determined by the presence of .argtypes attribute in the supplied function.

Parameters:

restype – default type of the function return value when calling wrap_bare() and restype is not supplied there explicitly (defaults to ctypes.int)
errcheck – default error-checking function which is automatically called for the return value; can also be overridden explicitly when calling wrapping methods if None, no error checking method
tuple_single_retval (bool) – determines if a single return values gets turned into a single-element tuple
return_res (bool) – determined if the function result gets returned; only used when list of return arguments (rvals) to wrapping functions is not explicitly supplied; can also be set to "auto" (default), which means that function returns its return value when no other rvals are found, and omits it otherwise.
default_rvals – default value for rvals in wrap_annotated() and wrap_bare(), if it is specified as None (default for those methods).
pointer_byref (bool) – if True, use explicit pointer creation instead of byref (in rare cases use of byref crashes the call).

byref(value)[source]¶

wrap_bare(func, argtypes, argnames=None, restype=None, args='nonrval', rvals='default', argprep=None, rconv=None, byref='all', errcheck=None)[source]¶

Annotate and wrap bare C function in a Python call.

Same as wrap_annotated(), but annotates the function first.

Parameters:

func – C function
argtypes – list of ctypes types corresponding to function arguments; gets assigned as func.argtypes
argnames – list of argument names; if not supplied, generated automatically as "arg1", "arg2", etc. Same for names which are defined as None.
restype – type of the function return value; if None, use the value supplied to the wrapper constructor (defaults to ctypes.int)
args – names of Python function arguments; can also be "all" (all C function arguments in that order), or "nonrval" (same, but with return value arguments excluded) by default, use "nonrval"
rvals – names of return value arguments; can include either a C function argument name, or None (which means the function return value); can also be "rest" (listsall the arguments not included into args; if args=="nonrval", assume that there are no rvals), "pointer" (assume that all pointer arguments are rvals; this does not include c_void_p, c_char_p, or c_wchar_p); by default, use the value supplied on the wrapper creation ("rest" by default)
argprep – dictionary {name: prep} of ways to prepare of C function arguments; each prep can be a value (which is assumed to be default argument value), or a callable, which is given values of Python function arguments
rconv – dictionary {name: conv} of converters of the return values; each conv is a function which takes 3 arguments: unconverted ctypes value, dictionary of all C function arguments, and dictionary of all Python function arguments if conv takes less than 3 argument, then the arguments list is trimmed (e.g., if it takes only one argument, it will be an unconverted value) conv can also be "ctypes" (return raw ctypes value), or "raw" (return raw value for buffers).
byref – list of all argument names which should by passed by reference; by default, it includes all arguments listed in rvals
errcheck – error-checking function which is automatically called for the return value; if None, use the value supplied to the wrapper constructor (none by default)

wrap_annotated(func, args='nonrval', rvals='default', alias=None, argprep=None, rconv=None, byref='all', errcheck=None)[source]¶

Wrap annotated C function in a Python call.

Assumes that the functions has defined .argtypes (list of argument types) and .argnames (list of argument names) attributes.

Parameters:

func – C function
args – names of Python function arguments; can also be "all" (all C function arguments in that order), or "nonrval" (same, but with return value arguments excluded); by default, use "nonrval"
rvals – names of return value arguments; can include either a C function argument name, or None (which means the function return value); can also be "rest" (listsall the arguments not included into args; if args=="nonrval", assume that there are no rvals), "pointer" (assume that all pointer arguments are rvals; this does not include c_void_p, c_char_p, or c_wchar_p); by default, use the value supplied on the wrapper creation ("rest" by default)
alias – either a list of argument names which replace .argnames, or a dictionary {argname: alias} which transforms names; all names in all other parameters (rvals, argprep, rconv, and byref) take aliased names
argprep – dictionary {name: prep} of ways to prepare of C function arguments; each prep can be a value (which is assumed to be default argument value), or a callable, which is given values of Python function arguments
rconv – dictionary {name: conv} of converters of the return values; each conv is a function which takes 3 arguments: unconverted ctypes value, dictionary of all C function arguments, and dictionary of all Python function arguments if conv takes less than 3 argument, then the arguments list is trimmed (e.g., if it takes only one argument, it will be an unconverted value)
byref – list of all argument names which should by passed by reference; by default, it includes all arguments listed in rvals
errcheck – error-checking function which is automatically called for the return value; if None, use the value supplied to the wrapper constructor (none by default)

pylablib.core.utils.ctypes_wrap.strprep(l, ctype=None, unicode=False)[source]¶

Make a string preparation function.

Return a function which creates a string with a fixed length of l bytes and returns a pointer to it. ctype can specify the type of the result (by default, ctypes.c_char_p).

pylablib.core.utils.ctypes_wrap.buffprep(size_arg_pos, dtype)[source]¶

Make a buffer preparation function.

Return a function which creates a string with a variable size (specified by an argument at a position size_arg_pos). The buffer size is given in elements. dtype specifies the datatype of the buffer, whose size is used to determine buffer size in bytes.

pylablib.core.utils.ctypes_wrap.buffconv(size_arg_pos, dtype)[source]¶

Make a buffer conversion function.

Return a function which converts a pointer of a variable size (specified by an argument at a position size_arg_pos) into a numpy array. The buffer size is given in elements. dtype specifies the datatype of the resulting array.

class pylablib.core.utils.ctypes_wrap.CStructWrapper(struct=None)[source]¶

Bases: object

Wrapper around a ctypes structure, which allows for easier creation of parsing of these structures.

When created, all structure fields can be accessed/modified as attributes of the wrapper object. It can also be converted into tuple using tup() method, or back into C structure using to_struct() method.

Class variable _struct should be set to the ctypes structure which is being wrapped. Several other class variables determine the behavior when generating and parsing:

_prep: dictionary {name: prep} of methods to prepare individual structure parameters; can be either a value or a function (which takes as ordered arguments all structure fields as ctypes values)

_conv: dictionary {name: conv} of methods to convert individual structure parameters when parsing a C structure; can be either a function (which takes ctypes value of the field as a single argument) or a value; also can be used as a source of default values on wrapper creation

_tup: dictionary {name: conv} of functions to convert structure values when generating a tuple

_tup_exc: list of values to exclude from the resulting tuple

_tup_inc: list of values to include in the resulting tuple (if None, include all)

_tup_add: list of values to add to the resulting tuple (these values must then exist either as attributes, or as entries in _tup dictionary)

_tup_order: order of fields in the returned tuple (by default, same as structure order)

Also specifies two overloaded methods for a more flexible preparation/conversion of structures. conv() takes no arguments and is called in the end of wrapper creation to finish setting up attributes. prep() takes a single argument (C structure) and is called when converting into a C structure to finish setting up the fields (e.g., size field).

Parameters:	struct – C structure to wrap (if `None`, create a new ‘blank’ structure).

to_struct()[source]¶: Convert wrapper into a C structure

prep(struct)[source]¶: Prepare C structure after creation (by default, do nothing)

conv()[source]¶: Prepare wrapper after setting up the fields from the wrapped structure

tup()[source]¶: Convert wrapper into a named tuple

classmethod prep_struct(*args, **kwargs)[source]¶: Prepare a blank C structure

classmethod prep_struct_args(*args, **kwargs)[source]¶: Prepare a C structure with the given supplied fields

classmethod tup_struct(struct, *args, **kwargs)[source]¶: Convert C structure into a named tuple

pylablib.core.utils.ctypes_wrap.class_tuple_to_dict(val, norm_strings=True, expand_lists=False)[source]¶

Convert a named tuple (usually, a tuple returned by CStructWrapper.tup()) into a dictionary.

Iterate recursively over all named tuple elements as well. If norm_strings==True, automatically translate byte strings into regular ones. If expand_lists==True, iterate recursively over lists members.

pylablib.core.utils.dictionary module¶

Tree-like multi-level dictionary with advanced indexing options.

pylablib.core.utils.dictionary.split_path(path, omit_empty=True, sep=None)[source]¶

Split generic path into individual path entries.

Parameters:	path – Generic path. Lists and tuples (possible nested) are flattened; strings are split according to separators; non-strings are converted into strings first. omit_empty (bool) – Determines if empty entries are skipped. sep (str) – If not `None`, defines regex for path separators; default separator is `'/'`.
Returns:	A list of individual entries.
Return type:	list

pylablib.core.utils.dictionary.normalize_path_entry(entry, case_normalization=None)[source]¶: Normalize the case of the entry if it’s not case-sensitive. Normalization is either None (no normalization, names are case-sensitive), 'lower' or 'upper'

pylablib.core.utils.dictionary.normalize_path(path, omit_empty=True, case_normalization=None, sep=None, force=False)[source]¶

Split and normalize generic path into individual path entries.

Parameters:	path – Generic path. Lists and tuples (possible nested) are flattened; strings are split according to separators; non-strings are converted into strings first. omit_empty (bool) – Determines if empty entries are skipped. case_normalization (str) – Case normalization rules; can be `None` (no normalization, names are case-sensitive), `'lower'` or `'upper'`. sep (str) – If not None, defines regex for path separators; default separator is `'/'`. force (bool) – If `False`, treat lists as if they’re already normalized.
Returns:	A list of individual normalized entries.
Return type:	list

pylablib.core.utils.dictionary.is_dictionary(obj, generic=False)[source]¶

Determine if the object is a dictionary.

Parameters:	obj – object generic (bool) – if `False`, passes only `Dictionary` (or subclasses) objects; otherwise, passes any dictionary-like object.
Returns:	bool

pylablib.core.utils.dictionary.as_dictionary(obj, case_normalization=None)[source]¶

Convert object into Dictionary with the given parameters.

If object is already a Dictionary (or its subclass), return unchanged, even if its parameters are different.

pylablib.core.utils.dictionary.as_dict(obj, style='nested', copy=True)[source]¶

Convert object into standard dict with the given parameters.

If object is already a dict, return unchanged, even if the parameters are different.

class pylablib.core.utils.dictionary.Dictionary(root=None, case_normalization=None, copy=True)[source]¶

Bases: object

Multi-level dictionary.

Access is done by path (all path elements are converted into strings and concatenated to form a single string path). If dictionary is not case-sensitive, all inserted and accessed paths are normalized to lower or upper case.

Parameters:	root (dict or Dictionary) – Initial value. case_normalization (str) – Case normalization rules; can be `None` (no normalization, names are case-sensitive), `'lower'` or `'upper'`. copy (bool) – If `True`, make copy of the supplied data; otherwise, just make it the root.

Warning

If copy==False, the root data is already assumed to be normalized. If it isn’t, the behavior might be incorrect.

static is_dictionary(obj, generic=True)[source]¶

Determine if the object is a dictionary.

Parameters:	obj – generic (bool) – if False, passes only `Dictionary` (or subclasses) objects; otherwise, passes any dictionary-like object.
Returns:	bool

static as_dictionary(obj, case_normalization=None)[source]¶

Convert object into Dictionary with the given parameters.

If object is already a Dictionary (or its subclass), return unchanged, even if its parameters are different.

add_entry(path, value, force=False, branch_option='normalize')[source]¶

Add value to a given path (overwrite leaf value if necessary).

Doesn’t replace leaves with branches and vice-verse if force==False.

Parameters:

path –
value –
force (bool) – If True, change leaf into a branch and vice-versa; otherwise, raises ValueError if the conversion is necessary.
branch_option (str) –
Decides what to do if the value is dictionary-like:
- 'attach' – just attach the root,
- 'copy' – copy and attach,
- 'normalize' – copy while normalizing all the keys according to the current rules.

get_entry(path, as_pointer=False)[source]¶

Get entry at a given path

Parameters:	path – as_pointer (bool) – If `True` and entry is not a leaf, return `DictionaryPointer`; otherwise, return `Dictionary`

has_entry(path, kind='all')[source]¶

Determine if the path is in the dictionary.

kind determines which kind of path to consider and can be 'leaf', 'branch' or 'all'.

is_leaf_path(path)[source]¶: Determine if the path is in the dictionary and points to a leaf

is_branch_path(path)[source]¶: Determine if the path is in the dictionary and points to a branch

get_max_prefix(path, kind='all')[source]¶

Find the longest prefix of path contained in the dictionary.

Return tuple (prefix, rest), where both path entries are normalized according to the dictionary rules (i.e., these are lists representing normalized paths). kind determines which kind of path to consider and can be 'leaf', 'branch' or 'all'. If the longest prefix is of a different kind, return (None,None).

del_entry(path)[source]¶

Delete entry from the dictionary.

Return True if the path was present. Note that it never raises KeyError.

size()[source]¶: Return the total size of the dictionary (number of nodes)

get(path, default=None)[source]¶: Analog of dict.get(): D.get(k,d) -> D[k] if k in D else d

pop(path, default=None)[source]¶

Analog of dict.pop(): remove value at path and return it if path in D, otherwise return default

Note that it never raises KeyError.

setdefault(path, default=None)[source]¶: Analog of dict.setdefault(): D.setdefault(k,d) -> D.get(k,d), also sets D[k]=d if k not in D.

items(ordered=False, leafs=False, path_kind='split', wrap_branches=True)[source]¶

Analog of dict.items(), by default iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

iteritems(ordered=False, leafs=False, path_kind='split', wrap_branches=True)¶

Analog of dict.items(), by default iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

viewitems(ordered=False, leafs=False, path_kind='split', wrap_branches=True)¶

Analog of dict.items(), by default iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

values(ordered=False, leafs=False, wrap_branches=True)[source]¶

Analog of dict.values(), iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

viewvalues(ordered=False, leafs=False, wrap_branches=True)¶

Analog of dict.values(), iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

itervalues(ordered=False, leafs=False, wrap_branches=True)¶

Analog of dict.values(), iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

keys(ordered=False, leafs=False, path_kind='split')[source]¶

Analog of dict.keys(), iterating only over the immediate children of the root.

Parameters:	ordered (bool) – If `True`, loop over keys in alphabetic order. leafs (bool) – If `True`, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary) path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

viewkeys(ordered=False, leafs=False, path_kind='split')¶

Analog of dict.keys(), iterating only over the immediate children of the root.

Parameters:	ordered (bool) – If `True`, loop over keys in alphabetic order. leafs (bool) – If `True`, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary) path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

iterkeys(ordered=False, leafs=False, path_kind='split')¶

Analog of dict.keys(), iterating only over the immediate children of the root.

Parameters:	ordered (bool) – If `True`, loop over keys in alphabetic order. leafs (bool) – If `True`, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary) path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

paths(ordered=False, topdown=False, path_kind='split')[source]¶

Return list of all paths (leafs and nodes).

Parameters:	ordered (bool) – If `True`, loop over paths in alphabetic order. topdown (bool) – If `True`, return node’s leafs before its subtrees leafs. path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

iternodes(to_visit='leafs', ordered=False, include_path=False, topdown=False)[source]¶

Iterate over nodes.

Parameters:	to_visit (str) – Can be `'leafs'`, `'branches'` or `'all'` and determines which parts of the dictionary are visited. ordered (bool) – If `True`, loop over paths in alphabetic order. include_path (bool) – Include in the return value. topdown (bool) – If `True`, visit node and its leafs before its subtrees leafs.
Yields:	Values for leafs and `DictionaryPointer` for branches. If `include_path==True`, yields tuple `(path, value)`, where path is in the form of a normalized list.

nodes(to_visit='leafs', ordered=False, include_path=False, topdown=False)¶

Iterate over nodes.

Parameters:	to_visit (str) – Can be `'leafs'`, `'branches'` or `'all'` and determines which parts of the dictionary are visited. ordered (bool) – If `True`, loop over paths in alphabetic order. include_path (bool) – Include in the return value. topdown (bool) – If `True`, visit node and its leafs before its subtrees leafs.
Yields:	Values for leafs and `DictionaryPointer` for branches. If `include_path==True`, yields tuple `(path, value)`, where path is in the form of a normalized list.

merge(source, path='', overwrite=True, normalize_paths=True)[source]¶

Attach source (dict or other Dictionary) to a given branch; source is automatically deep-copied.

If source is not a dictionary, simply assign it (i.e., D.merge(v,p) is equivalent to D.add_entry(p,v,force=True) in this case). Compared to add_entry(), merges two branches instead of removing the old branch completely.

Parameters:	source (dict or Dictionary) – branch (tuple or str) – Destination path. overwrite (bool) – If `True`, replaces the old entries with the new ones (it only matters for leaf assignments). normalize_paths (bool) – If `True` and the dictionary isn’t case sensitive, perform normalization if the source.

update(source, path='', overwrite=True, normalize_paths=True)¶

Attach source (dict or other Dictionary) to a given branch; source is automatically deep-copied.

If source is not a dictionary, simply assign it (i.e., D.merge(v,p) is equivalent to D.add_entry(p,v,force=True) in this case). Compared to add_entry(), merges two branches instead of removing the old branch completely.

Parameters:	source (dict or Dictionary) – branch (tuple or str) – Destination path. overwrite (bool) – If `True`, replaces the old entries with the new ones (it only matters for leaf assignments). normalize_paths (bool) – If `True` and the dictionary isn’t case sensitive, perform normalization if the source.

detach(path)[source]¶

Remove a branch or a leaf from the current dictionary.

Branch is returned as a separate Dictionary. If path is missing, raise a KeyError.

collect(paths, detach=False, ignore_missing=True)[source]¶

Collect a set of subpaths into a separate dictionary.

Parameters:	paths – list or set of paths detach – if `True`, added branches are removed from this dictionary ignore_missing – if `True`, ignore paths from the list which are not present in this dictionary; otherwise, raise a `KeyError`.

branch_copy(branch='')[source]¶: Get a copy of the branch as a Dictionary

copy()[source]¶: Get a full copy the dictionary

updated(source, path='', overwrite=True, normalize_paths=True)[source]¶

Get a copy of the dictionary and attach a new branch to it.

Parameters are the same as in the Dictionary.merge().

as_dict(style='nested', copy=True)[source]¶

Convert into a dict object.

Parameters:	style (str) – Determines style of the result: `'nested'` – subtrees are turned into nested dictionaries, `'flat'` – single dictionary is formed with full paths as keys. copy (bool) – If `False` and `style=='nested'`, return the root dictionary.

asdict(style='nested', copy=True)¶

Convert into a dict object.

Parameters:	style (str) – Determines style of the result: `'nested'` – subtrees are turned into nested dictionaries, `'flat'` – single dictionary is formed with full paths as keys. copy (bool) – If `False` and `style=='nested'`, return the root dictionary.

as_json(style='nested')[source]¶

Convert into a JSON string.

Parameters:	style (str) – Determines style of the result: - `'nested'` – subtrees are turned into nested dictionaries, - `'flat'` – single dictionary is formed with full paths as keys.

classmethod from_json(data, case_normalization=None)[source]¶: Convert JSON representations of a dictionary into a Dictionary object

as_pandas(index_key=True, as_series=True)[source]¶

Convert into a pandas DataFrame or Series object.

Parameters:	index_key (bool) – If `False`, create a 2-column table with the first column (`"key"`) containing string path and the second column (`"value"`) containing value; otherwise, move key to the table index. as_series (bool) – If `index_key==True` and `as_series==True`, convert the resulting DataFrame into 1D Series (the key is the index); otherwise, keep it as a single-column table

get_path()[source]¶

branch_pointer(branch='')[source]¶: Get a DictionaryPointer of a given branch

map_self(func, to_visit='leafs', pass_path=False, topdown=False, branch_option='normalize')[source]¶

Apply func to the nodes in the dictionary.

Note that any pointers to the replaced branches or their sub-branches will become invalid.

Parameters:

func (callable) – Mapping function. Leafs are passed by value, branches (if visited) are passed as DictionaryPointer.
to_visit (str) – Can be 'leafs', 'branches' or 'all' and determines which parts of the dictionary passed to the map function.
pass_path (bool) – If True, pass the node path (in the form of a normalized list) as a first argument to func.
topdown (bool) – If True, visit node and its leafs before its subtrees leafs.
branch_option (str) – If the function returns a dict-like object, determines how to incorporate into the dictionary; can be "normalize" (make a copy with normalized paths and insert that), "copy" (make a copy without normalization), or "attach" (simply replace the value without copying and normalization)

filter_self(pred, to_visit='leafs', pass_path=False, topdown=False)[source]¶

Remove all the nodes from the dictionary for which pred returns False.

Parameters:

pred (callable) – Filter function. Leafs are passed to pred by value, branches (if visited) are passed as DictionaryPointer.
to_visit (str) – Can be 'leafs', 'branches' or 'all' and determines which parts of the dictionary passed to the predicate.
pass_path (bool) – If True, pass the node path (in the form of a normalized list) as a first argument to pred.
topdown (bool) – If True, visit node and its leafs before its subtrees leafs.

diff(other)[source]¶

Perform an element-wise comparison to another Dictionary.

If the other Dictionary has a different case sensitivity, raise ValueError.

Returns:	`DictionaryDiff`

static diff_flatdict(first, second)[source]¶

Find the difference between flat dict objects.

Returns:	`DictionaryDiff`

static find_intersection(dicts, use_flatten=False)[source]¶

Find intersection of multiple dictionaries.

Parameters:	dicts ([`Dictionary`]) – use_flatten (bool) – If `True` flatten all dictionaries before comparison (works faster for a large number of dictionaries).
Returns:	`DictionaryIntersection`

get_matching_paths(pattern, wildkey='*', wildpath='**', only_leaves=True)[source]¶

Get all paths in the tree that match the provided pattern.

Parameters:	wildkey (str) – Pattern symbol that matches any key. wildpath (str) – Pattern symbol that matches any subpath (possibly empty). only_leaves (bool) – If `True`, only check leaf paths; otherwise, check subtree paths (i.e., incomplete leaf paths) as well. Basically, `only_leaves=False` is analogous to adding wildpath at the end of the pattern.

get_matching_subtree(pattern, wildkey='*', wildpath='**', only_leaves=True)[source]¶

Get a subtree containing nodes with paths matching the provided pattern.

Parameters:	wildkey (str) – Pattern symbol that matches any key. wildpath (str) – Pattern symbol that matches any subpath (possibly empty). only_leaves (bool) – If `True`, only check leaf paths; otherwise, check subtree paths (i.e., incomplete leaf paths) as well. Basically, `only_leaves=False` is analogous to adding wildpath at the end of the pattern.

class pylablib.core.utils.dictionary.DictionaryDiff[source]¶

Bases: pylablib.core.utils.dictionary.DictionaryDiff

Describes a difference between the two dictionaries.

same¶

Contains the leafs which is the same.

Type:	`Dictionary`

changed_from¶

Contains the leafs from the first dictionary which have different values in the second dictionary.

Type:	`Dictionary`

changed_to¶

Contains the leafs from the second dictionary which have different values in the first dictionary.

Type:	`Dictionary`

removed¶

Contains the leafs from the first dictionary which are absent in the second dictionary.

Type:	`Dictionary`

added¶

Contains the leafs from the second dictionary which are absent in the first dictionary.

Type:	`Dictionary`

added

changed_from

changed_to

count()¶: Return number of occurrences of value.

index()¶

Return first index of value.

Raises ValueError if the value is not present.

removed

same

class pylablib.core.utils.dictionary.DictionaryIntersection[source]¶

Bases: pylablib.core.utils.dictionary.DictionaryIntersection

Describes the result of finding intersection of multiple dictionaries.

common¶

Contains the intersection of all dictionaries.

Type:	`Dictionary`

individual¶

Contains list of difference from intersection for all dictionaries.

Type:	[`Dictionary`]

common

count()¶: Return number of occurrences of value.

index()¶

Return first index of value.

Raises ValueError if the value is not present.

individual

class pylablib.core.utils.dictionary.DictionaryPointer(root=None, pointer=None, case_normalization=None, copy=True)[source]¶

Bases: pylablib.core.utils.dictionary.Dictionary

Similar to Dictionary, but can point at one of the branches instead of the full dictionary.

Effect is mostly equivalent to prepending some path to all queries.

Parameters:	root (dict or Dictionary) – Complete tree. pointer – Path to the pointer location. case_normalization (str) – Case normalization rules; can be `None` (no normalization, names are case-sensitive), `'lower'` or `'upper'`. copy (bool) – If `True`, make copy of the supplied data; otherwise, just make it the root.

Warning

If copy==False, the root data is already assumed to be normalized. If it isn’t, the behavior might be incorrect.

get_path()[source]¶: Return pointer path in the whole dictionary.

move_to(path='', absolute=True)[source]¶

Move the pointer to a new path.

Parameters:	path – absolute (bool) – If `True`, path is specified with respect to the root; otherwise, it’s specified with respect to the current position (and can only go deeper).

move_up(levels, strict=True)[source]¶

Move the pointer by the given number of levels up.

If strict==True and there are not enough levels above, raise an error. Otherwise, stop at the top dictionary level.

branch_pointer(branch='')[source]¶: Get a DictionaryPointer of a given branch.

add_entry(path, value, force=False, branch_option='normalize')¶

Add value to a given path (overwrite leaf value if necessary).

Doesn’t replace leaves with branches and vice-verse if force==False.

Parameters:

path –
value –
force (bool) – If True, change leaf into a branch and vice-versa; otherwise, raises ValueError if the conversion is necessary.
branch_option (str) –
Decides what to do if the value is dictionary-like:
- 'attach' – just attach the root,
- 'copy' – copy and attach,
- 'normalize' – copy while normalizing all the keys according to the current rules.

as_dict(style='nested', copy=True)¶

Convert into a dict object.

Parameters:	style (str) – Determines style of the result: `'nested'` – subtrees are turned into nested dictionaries, `'flat'` – single dictionary is formed with full paths as keys. copy (bool) – If `False` and `style=='nested'`, return the root dictionary.

static as_dictionary(obj, case_normalization=None)¶

Convert object into Dictionary with the given parameters.

If object is already a Dictionary (or its subclass), return unchanged, even if its parameters are different.

as_json(style='nested')¶

Convert into a JSON string.

Parameters:	style (str) – Determines style of the result: - `'nested'` – subtrees are turned into nested dictionaries, - `'flat'` – single dictionary is formed with full paths as keys.

as_pandas(index_key=True, as_series=True)¶

Convert into a pandas DataFrame or Series object.

Parameters:	index_key (bool) – If `False`, create a 2-column table with the first column (`"key"`) containing string path and the second column (`"value"`) containing value; otherwise, move key to the table index. as_series (bool) – If `index_key==True` and `as_series==True`, convert the resulting DataFrame into 1D Series (the key is the index); otherwise, keep it as a single-column table

asdict(style='nested', copy=True)¶

Convert into a dict object.

Parameters:	style (str) – Determines style of the result: `'nested'` – subtrees are turned into nested dictionaries, `'flat'` – single dictionary is formed with full paths as keys. copy (bool) – If `False` and `style=='nested'`, return the root dictionary.

branch_copy(branch='')¶: Get a copy of the branch as a Dictionary

collect(paths, detach=False, ignore_missing=True)¶

Collect a set of subpaths into a separate dictionary.

Parameters:	paths – list or set of paths detach – if `True`, added branches are removed from this dictionary ignore_missing – if `True`, ignore paths from the list which are not present in this dictionary; otherwise, raise a `KeyError`.

copy()¶: Get a full copy the dictionary

del_entry(path)¶

Delete entry from the dictionary.

Return True if the path was present. Note that it never raises KeyError.

detach(path)¶

Remove a branch or a leaf from the current dictionary.

Branch is returned as a separate Dictionary. If path is missing, raise a KeyError.

diff(other)¶

Perform an element-wise comparison to another Dictionary.

If the other Dictionary has a different case sensitivity, raise ValueError.

Returns:	`DictionaryDiff`

static diff_flatdict(first, second)¶

Find the difference between flat dict objects.

Returns:	`DictionaryDiff`

filter_self(pred, to_visit='leafs', pass_path=False, topdown=False)¶

Remove all the nodes from the dictionary for which pred returns False.

Parameters:

pred (callable) – Filter function. Leafs are passed to pred by value, branches (if visited) are passed as DictionaryPointer.
to_visit (str) – Can be 'leafs', 'branches' or 'all' and determines which parts of the dictionary passed to the predicate.
pass_path (bool) – If True, pass the node path (in the form of a normalized list) as a first argument to pred.
topdown (bool) – If True, visit node and its leafs before its subtrees leafs.

static find_intersection(dicts, use_flatten=False)¶

Find intersection of multiple dictionaries.

Parameters:	dicts ([`Dictionary`]) – use_flatten (bool) – If `True` flatten all dictionaries before comparison (works faster for a large number of dictionaries).
Returns:	`DictionaryIntersection`

classmethod from_json(data, case_normalization=None)¶: Convert JSON representations of a dictionary into a Dictionary object

get(path, default=None)¶: Analog of dict.get(): D.get(k,d) -> D[k] if k in D else d

get_entry(path, as_pointer=False)¶

Get entry at a given path

Parameters:	path – as_pointer (bool) – If `True` and entry is not a leaf, return `DictionaryPointer`; otherwise, return `Dictionary`

get_matching_paths(pattern, wildkey='*', wildpath='**', only_leaves=True)¶

Get all paths in the tree that match the provided pattern.

Parameters:	wildkey (str) – Pattern symbol that matches any key. wildpath (str) – Pattern symbol that matches any subpath (possibly empty). only_leaves (bool) – If `True`, only check leaf paths; otherwise, check subtree paths (i.e., incomplete leaf paths) as well. Basically, `only_leaves=False` is analogous to adding wildpath at the end of the pattern.

get_matching_subtree(pattern, wildkey='*', wildpath='**', only_leaves=True)¶

Get a subtree containing nodes with paths matching the provided pattern.

Parameters:	wildkey (str) – Pattern symbol that matches any key. wildpath (str) – Pattern symbol that matches any subpath (possibly empty). only_leaves (bool) – If `True`, only check leaf paths; otherwise, check subtree paths (i.e., incomplete leaf paths) as well. Basically, `only_leaves=False` is analogous to adding wildpath at the end of the pattern.

get_max_prefix(path, kind='all')¶

Find the longest prefix of path contained in the dictionary.

Return tuple (prefix, rest), where both path entries are normalized according to the dictionary rules (i.e., these are lists representing normalized paths). kind determines which kind of path to consider and can be 'leaf', 'branch' or 'all'. If the longest prefix is of a different kind, return (None,None).

has_entry(path, kind='all')¶

Determine if the path is in the dictionary.

kind determines which kind of path to consider and can be 'leaf', 'branch' or 'all'.

is_branch_path(path)¶: Determine if the path is in the dictionary and points to a branch

static is_dictionary(obj, generic=True)¶

Determine if the object is a dictionary.

Parameters:	obj – generic (bool) – if False, passes only `Dictionary` (or subclasses) objects; otherwise, passes any dictionary-like object.
Returns:	bool

is_leaf_path(path)¶: Determine if the path is in the dictionary and points to a leaf

items(ordered=False, leafs=False, path_kind='split', wrap_branches=True)¶

Analog of dict.items(), by default iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

iteritems(ordered=False, leafs=False, path_kind='split', wrap_branches=True)¶

Analog of dict.items(), by default iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

iterkeys(ordered=False, leafs=False, path_kind='split')¶

Analog of dict.keys(), iterating only over the immediate children of the root.

Parameters:	ordered (bool) – If `True`, loop over keys in alphabetic order. leafs (bool) – If `True`, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary) path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

iternodes(to_visit='leafs', ordered=False, include_path=False, topdown=False)¶

Iterate over nodes.

Parameters:	to_visit (str) – Can be `'leafs'`, `'branches'` or `'all'` and determines which parts of the dictionary are visited. ordered (bool) – If `True`, loop over paths in alphabetic order. include_path (bool) – Include in the return value. topdown (bool) – If `True`, visit node and its leafs before its subtrees leafs.
Yields:	Values for leafs and `DictionaryPointer` for branches. If `include_path==True`, yields tuple `(path, value)`, where path is in the form of a normalized list.

itervalues(ordered=False, leafs=False, wrap_branches=True)¶

Analog of dict.values(), iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

keys(ordered=False, leafs=False, path_kind='split')¶

Analog of dict.keys(), iterating only over the immediate children of the root.

Parameters:	ordered (bool) – If `True`, loop over keys in alphabetic order. leafs (bool) – If `True`, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary) path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

map_self(func, to_visit='leafs', pass_path=False, topdown=False, branch_option='normalize')¶

Apply func to the nodes in the dictionary.

Note that any pointers to the replaced branches or their sub-branches will become invalid.

Parameters:

func (callable) – Mapping function. Leafs are passed by value, branches (if visited) are passed as DictionaryPointer.
to_visit (str) – Can be 'leafs', 'branches' or 'all' and determines which parts of the dictionary passed to the map function.
pass_path (bool) – If True, pass the node path (in the form of a normalized list) as a first argument to func.
topdown (bool) – If True, visit node and its leafs before its subtrees leafs.
branch_option (str) – If the function returns a dict-like object, determines how to incorporate into the dictionary; can be "normalize" (make a copy with normalized paths and insert that), "copy" (make a copy without normalization), or "attach" (simply replace the value without copying and normalization)

merge(source, path='', overwrite=True, normalize_paths=True)¶

Attach source (dict or other Dictionary) to a given branch; source is automatically deep-copied.

If source is not a dictionary, simply assign it (i.e., D.merge(v,p) is equivalent to D.add_entry(p,v,force=True) in this case). Compared to add_entry(), merges two branches instead of removing the old branch completely.

Parameters:	source (dict or Dictionary) – branch (tuple or str) – Destination path. overwrite (bool) – If `True`, replaces the old entries with the new ones (it only matters for leaf assignments). normalize_paths (bool) – If `True` and the dictionary isn’t case sensitive, perform normalization if the source.

nodes(to_visit='leafs', ordered=False, include_path=False, topdown=False)¶

Iterate over nodes.

Parameters:	to_visit (str) – Can be `'leafs'`, `'branches'` or `'all'` and determines which parts of the dictionary are visited. ordered (bool) – If `True`, loop over paths in alphabetic order. include_path (bool) – Include in the return value. topdown (bool) – If `True`, visit node and its leafs before its subtrees leafs.
Yields:	Values for leafs and `DictionaryPointer` for branches. If `include_path==True`, yields tuple `(path, value)`, where path is in the form of a normalized list.

paths(ordered=False, topdown=False, path_kind='split')¶

Return list of all paths (leafs and nodes).

Parameters:	ordered (bool) – If `True`, loop over paths in alphabetic order. topdown (bool) – If `True`, return node’s leafs before its subtrees leafs. path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

pop(path, default=None)¶

Analog of dict.pop(): remove value at path and return it if path in D, otherwise return default

Note that it never raises KeyError.

setdefault(path, default=None)¶: Analog of dict.setdefault(): D.setdefault(k,d) -> D.get(k,d), also sets D[k]=d if k not in D.

size()¶: Return the total size of the dictionary (number of nodes)

update(source, path='', overwrite=True, normalize_paths=True)¶

Attach source (dict or other Dictionary) to a given branch; source is automatically deep-copied.

If source is not a dictionary, simply assign it (i.e., D.merge(v,p) is equivalent to D.add_entry(p,v,force=True) in this case). Compared to add_entry(), merges two branches instead of removing the old branch completely.

Parameters:	source (dict or Dictionary) – branch (tuple or str) – Destination path. overwrite (bool) – If `True`, replaces the old entries with the new ones (it only matters for leaf assignments). normalize_paths (bool) – If `True` and the dictionary isn’t case sensitive, perform normalization if the source.

updated(source, path='', overwrite=True, normalize_paths=True)¶

Get a copy of the dictionary and attach a new branch to it.

Parameters are the same as in the Dictionary.merge().

values(ordered=False, leafs=False, wrap_branches=True)¶

Analog of dict.values(), iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

viewitems(ordered=False, leafs=False, path_kind='split', wrap_branches=True)¶

Analog of dict.items(), by default iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

viewkeys(ordered=False, leafs=False, path_kind='split')¶

Analog of dict.keys(), iterating only over the immediate children of the root.

Parameters:	ordered (bool) – If `True`, loop over keys in alphabetic order. leafs (bool) – If `True`, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary) path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

viewvalues(ordered=False, leafs=False, wrap_branches=True)¶

Analog of dict.values(), iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

pylablib.core.utils.dictionary.combine_dictionaries(dicts, func, select='all', pass_missing=False)[source]¶

Combine several dictionaries element-wise (only for leafs) using a given function.

Parameters:

dicts (list or tuple) – list of dictionaries (Dictionary or dict) to be combined
func (callable) – combination function. Takes a single argument, which is a list of elements to be combined.
select (str) – determines which keys are selected for the resulting dictionary. Can be either "all" (only keep keys which are present in all the dictionaries), or "any" (keep keys which are present in at least one dictionary). Only keys that point to leafs count; if a key points to a non-leaf branch in some dictionary, it is considered absent from this dictionary.
pass_missing (bool) – if select=="any", this parameter determines whether missing elements will be passed to func as None, or omitted entirely.

class pylablib.core.utils.dictionary.PrefixTree(root=None, case_normalization=None, wildcard='*', matchcard='.', copy=True)[source]¶

Bases: pylablib.core.utils.dictionary.Dictionary

Expansion of a Dictionary designed to store data related to prefixes.

Each branch node can have a leaf with a name given by wildcard ('*' by default) or matchcard ('.' by default). Wildcard assumes that the branch node path is a prefix; matchcard assumes exact match. These leafs are inspected when specific prefix tree functions (find_largest_prefix() and find_all_prefixes()) are used.

Parameters:

root (dict or Dictionary) – Complete tree.
case_normalization (str) – Case normalization rules; can be None (no normalization, names are case-sensitive), 'lower' or 'upper'.
wildcard (str) – Symbol for a wildcard entry.
matchcard (str) – Symbol for a matchcard entry.
copy (bool) – If True, make copy of the supplied data; otherwise, just make it the root.

Warning

If copy==False, the root data is already assumed to be normalized. If it isn’t, the behavior might be incorrect.

copy()[source]¶: Get a full copy the prefix tree

find_largest_prefix(path, default=None, allow_nomatch_exact=True, return_path=False, return_subpath=False)[source]¶

Find the entry which is the largest prefix of a given path.

Parameters:

path –
default – Default value if the path isn’t found.
allow_nomatch_exact (bool) – If True, just element with the given path can be returned; otherwise, only elements stored under wildcards and matchcards are considered.
return_path (bool) – If True, return path to the element (i.e., the largest prefix) instead of the element itself.
return_subpath (bool) – If True, return tuple with a second element being part of the path left after subtraction of the prefix.

find_all_prefixes(path, allow_nomatch_exact=True, return_path=True, return_subpath=False)[source]¶

Find list of all the entries which are prefixes of a given path.

Parameters:

path –
default – Default value if the path isn’t found.
allow_nomatch_exact (bool) – If True, just element with the given path can be returned; otherwise, only elements stored under wildcards and matchcards are considered.
return_path (bool) – If True, return path to the element (i.e., the largest prefix) instead of the element itself.
return_subpath (bool) – If True, return tuple with a second element being part of the path left after subtraction of the prefix.

add_entry(path, value, force=False, branch_option='normalize')¶

Add value to a given path (overwrite leaf value if necessary).

Doesn’t replace leaves with branches and vice-verse if force==False.

Parameters:

path –
value –
force (bool) – If True, change leaf into a branch and vice-versa; otherwise, raises ValueError if the conversion is necessary.
branch_option (str) –
Decides what to do if the value is dictionary-like:
- 'attach' – just attach the root,
- 'copy' – copy and attach,
- 'normalize' – copy while normalizing all the keys according to the current rules.

as_dict(style='nested', copy=True)¶

Convert into a dict object.

Parameters:	style (str) – Determines style of the result: `'nested'` – subtrees are turned into nested dictionaries, `'flat'` – single dictionary is formed with full paths as keys. copy (bool) – If `False` and `style=='nested'`, return the root dictionary.

static as_dictionary(obj, case_normalization=None)¶

Convert object into Dictionary with the given parameters.

If object is already a Dictionary (or its subclass), return unchanged, even if its parameters are different.

as_json(style='nested')¶

Convert into a JSON string.

Parameters:	style (str) – Determines style of the result: - `'nested'` – subtrees are turned into nested dictionaries, - `'flat'` – single dictionary is formed with full paths as keys.

as_pandas(index_key=True, as_series=True)¶

Convert into a pandas DataFrame or Series object.

Parameters:	index_key (bool) – If `False`, create a 2-column table with the first column (`"key"`) containing string path and the second column (`"value"`) containing value; otherwise, move key to the table index. as_series (bool) – If `index_key==True` and `as_series==True`, convert the resulting DataFrame into 1D Series (the key is the index); otherwise, keep it as a single-column table

asdict(style='nested', copy=True)¶

Convert into a dict object.

Parameters:	style (str) – Determines style of the result: `'nested'` – subtrees are turned into nested dictionaries, `'flat'` – single dictionary is formed with full paths as keys. copy (bool) – If `False` and `style=='nested'`, return the root dictionary.

branch_copy(branch='')¶: Get a copy of the branch as a Dictionary

branch_pointer(branch='')¶: Get a DictionaryPointer of a given branch

collect(paths, detach=False, ignore_missing=True)¶

Collect a set of subpaths into a separate dictionary.

Parameters:	paths – list or set of paths detach – if `True`, added branches are removed from this dictionary ignore_missing – if `True`, ignore paths from the list which are not present in this dictionary; otherwise, raise a `KeyError`.

del_entry(path)¶

Delete entry from the dictionary.

Return True if the path was present. Note that it never raises KeyError.

detach(path)¶

Remove a branch or a leaf from the current dictionary.

Branch is returned as a separate Dictionary. If path is missing, raise a KeyError.

diff(other)¶

Perform an element-wise comparison to another Dictionary.

If the other Dictionary has a different case sensitivity, raise ValueError.

Returns:	`DictionaryDiff`

static diff_flatdict(first, second)¶

Find the difference between flat dict objects.

Returns:	`DictionaryDiff`

filter_self(pred, to_visit='leafs', pass_path=False, topdown=False)¶

Remove all the nodes from the dictionary for which pred returns False.

Parameters:

pred (callable) – Filter function. Leafs are passed to pred by value, branches (if visited) are passed as DictionaryPointer.
to_visit (str) – Can be 'leafs', 'branches' or 'all' and determines which parts of the dictionary passed to the predicate.
pass_path (bool) – If True, pass the node path (in the form of a normalized list) as a first argument to pred.
topdown (bool) – If True, visit node and its leafs before its subtrees leafs.

static find_intersection(dicts, use_flatten=False)¶

Find intersection of multiple dictionaries.

Parameters:	dicts ([`Dictionary`]) – use_flatten (bool) – If `True` flatten all dictionaries before comparison (works faster for a large number of dictionaries).
Returns:	`DictionaryIntersection`

classmethod from_json(data, case_normalization=None)¶: Convert JSON representations of a dictionary into a Dictionary object

get(path, default=None)¶: Analog of dict.get(): D.get(k,d) -> D[k] if k in D else d

get_entry(path, as_pointer=False)¶

Get entry at a given path

Parameters:	path – as_pointer (bool) – If `True` and entry is not a leaf, return `DictionaryPointer`; otherwise, return `Dictionary`

get_matching_paths(pattern, wildkey='*', wildpath='**', only_leaves=True)¶

Get all paths in the tree that match the provided pattern.

Parameters:	wildkey (str) – Pattern symbol that matches any key. wildpath (str) – Pattern symbol that matches any subpath (possibly empty). only_leaves (bool) – If `True`, only check leaf paths; otherwise, check subtree paths (i.e., incomplete leaf paths) as well. Basically, `only_leaves=False` is analogous to adding wildpath at the end of the pattern.

get_matching_subtree(pattern, wildkey='*', wildpath='**', only_leaves=True)¶

Get a subtree containing nodes with paths matching the provided pattern.

Parameters:	wildkey (str) – Pattern symbol that matches any key. wildpath (str) – Pattern symbol that matches any subpath (possibly empty). only_leaves (bool) – If `True`, only check leaf paths; otherwise, check subtree paths (i.e., incomplete leaf paths) as well. Basically, `only_leaves=False` is analogous to adding wildpath at the end of the pattern.

get_max_prefix(path, kind='all')¶

Find the longest prefix of path contained in the dictionary.

Return tuple (prefix, rest), where both path entries are normalized according to the dictionary rules (i.e., these are lists representing normalized paths). kind determines which kind of path to consider and can be 'leaf', 'branch' or 'all'. If the longest prefix is of a different kind, return (None,None).

get_path()¶

has_entry(path, kind='all')¶

Determine if the path is in the dictionary.

kind determines which kind of path to consider and can be 'leaf', 'branch' or 'all'.

is_branch_path(path)¶: Determine if the path is in the dictionary and points to a branch

static is_dictionary(obj, generic=True)¶

Determine if the object is a dictionary.

Parameters:	obj – generic (bool) – if False, passes only `Dictionary` (or subclasses) objects; otherwise, passes any dictionary-like object.
Returns:	bool

is_leaf_path(path)¶: Determine if the path is in the dictionary and points to a leaf

items(ordered=False, leafs=False, path_kind='split', wrap_branches=True)¶

Analog of dict.items(), by default iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

iteritems(ordered=False, leafs=False, path_kind='split', wrap_branches=True)¶

Analog of dict.items(), by default iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

iterkeys(ordered=False, leafs=False, path_kind='split')¶

Analog of dict.keys(), iterating only over the immediate children of the root.

Parameters:	ordered (bool) – If `True`, loop over keys in alphabetic order. leafs (bool) – If `True`, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary) path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

iternodes(to_visit='leafs', ordered=False, include_path=False, topdown=False)¶

Iterate over nodes.

Parameters:	to_visit (str) – Can be `'leafs'`, `'branches'` or `'all'` and determines which parts of the dictionary are visited. ordered (bool) – If `True`, loop over paths in alphabetic order. include_path (bool) – Include in the return value. topdown (bool) – If `True`, visit node and its leafs before its subtrees leafs.
Yields:	Values for leafs and `DictionaryPointer` for branches. If `include_path==True`, yields tuple `(path, value)`, where path is in the form of a normalized list.

itervalues(ordered=False, leafs=False, wrap_branches=True)¶

Analog of dict.values(), iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

keys(ordered=False, leafs=False, path_kind='split')¶

Analog of dict.keys(), iterating only over the immediate children of the root.

Parameters:	ordered (bool) – If `True`, loop over keys in alphabetic order. leafs (bool) – If `True`, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary) path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

map_self(func, to_visit='leafs', pass_path=False, topdown=False, branch_option='normalize')¶

Apply func to the nodes in the dictionary.

Note that any pointers to the replaced branches or their sub-branches will become invalid.

Parameters:

func (callable) – Mapping function. Leafs are passed by value, branches (if visited) are passed as DictionaryPointer.
to_visit (str) – Can be 'leafs', 'branches' or 'all' and determines which parts of the dictionary passed to the map function.
pass_path (bool) – If True, pass the node path (in the form of a normalized list) as a first argument to func.
topdown (bool) – If True, visit node and its leafs before its subtrees leafs.
branch_option (str) – If the function returns a dict-like object, determines how to incorporate into the dictionary; can be "normalize" (make a copy with normalized paths and insert that), "copy" (make a copy without normalization), or "attach" (simply replace the value without copying and normalization)

merge(source, path='', overwrite=True, normalize_paths=True)¶

Attach source (dict or other Dictionary) to a given branch; source is automatically deep-copied.

If source is not a dictionary, simply assign it (i.e., D.merge(v,p) is equivalent to D.add_entry(p,v,force=True) in this case). Compared to add_entry(), merges two branches instead of removing the old branch completely.

Parameters:	source (dict or Dictionary) – branch (tuple or str) – Destination path. overwrite (bool) – If `True`, replaces the old entries with the new ones (it only matters for leaf assignments). normalize_paths (bool) – If `True` and the dictionary isn’t case sensitive, perform normalization if the source.

nodes(to_visit='leafs', ordered=False, include_path=False, topdown=False)¶

Iterate over nodes.

Parameters:	to_visit (str) – Can be `'leafs'`, `'branches'` or `'all'` and determines which parts of the dictionary are visited. ordered (bool) – If `True`, loop over paths in alphabetic order. include_path (bool) – Include in the return value. topdown (bool) – If `True`, visit node and its leafs before its subtrees leafs.
Yields:	Values for leafs and `DictionaryPointer` for branches. If `include_path==True`, yields tuple `(path, value)`, where path is in the form of a normalized list.

paths(ordered=False, topdown=False, path_kind='split')¶

Return list of all paths (leafs and nodes).

Parameters:	ordered (bool) – If `True`, loop over paths in alphabetic order. topdown (bool) – If `True`, return node’s leafs before its subtrees leafs. path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

pop(path, default=None)¶

Analog of dict.pop(): remove value at path and return it if path in D, otherwise return default

Note that it never raises KeyError.

setdefault(path, default=None)¶: Analog of dict.setdefault(): D.setdefault(k,d) -> D.get(k,d), also sets D[k]=d if k not in D.

size()¶: Return the total size of the dictionary (number of nodes)

update(source, path='', overwrite=True, normalize_paths=True)¶

Attach source (dict or other Dictionary) to a given branch; source is automatically deep-copied.

If source is not a dictionary, simply assign it (i.e., D.merge(v,p) is equivalent to D.add_entry(p,v,force=True) in this case). Compared to add_entry(), merges two branches instead of removing the old branch completely.

Parameters:	source (dict or Dictionary) – branch (tuple or str) – Destination path. overwrite (bool) – If `True`, replaces the old entries with the new ones (it only matters for leaf assignments). normalize_paths (bool) – If `True` and the dictionary isn’t case sensitive, perform normalization if the source.

updated(source, path='', overwrite=True, normalize_paths=True)¶

Get a copy of the dictionary and attach a new branch to it.

Parameters are the same as in the Dictionary.merge().

values(ordered=False, leafs=False, wrap_branches=True)¶

Analog of dict.values(), iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

viewitems(ordered=False, leafs=False, path_kind='split', wrap_branches=True)¶

Analog of dict.items(), by default iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

viewkeys(ordered=False, leafs=False, path_kind='split')¶

Analog of dict.keys(), iterating only over the immediate children of the root.

Parameters:	ordered (bool) – If `True`, loop over keys in alphabetic order. leafs (bool) – If `True`, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary) path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

viewvalues(ordered=False, leafs=False, wrap_branches=True)¶

Analog of dict.values(), iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

class pylablib.core.utils.dictionary.FilterTree(root=None, case_normalization=None, default=False, match_prefix=False, copy=True)[source]¶

Bases: pylablib.core.utils.dictionary.Dictionary

Expansion of a Dictionary designed to store hierarchical path filtering rules.

Store path templates and the corresponding values (usually True or False for a filter tree, but other values are possible). The match() method is then tested against this templates, and the value of the closest matching template (or default value, if none match) is returned. The templates can contain direct matches (e.g., "a/b/c", which matches only "a/b/c/" path), "*" path entries for a single level wildcard (e.g., "a/*/c" matches "a/b/c" or 'a/d/c", but not "a/c" or "a/b/d/c"), or "**" path entries for a multi-level wildcard (e.g., "a/**/c" matches "a/b/c", "a/c", or "a/b/d/c"). The paths are always tested first for direct match, then for "*" match, then for "**" match starting from the smallest subpath matching "**".

Parameters:

root (dict or Dictionary) – A filter tree or a list of filter tree paths (which are all assumed to be have the True value).s
case_normalization (str) – Case normalization rules; can be None (no normalization, names are case-sensitive), 'lower' or 'upper'.
default – Default value to return if no match is found.
match_prefix – if True, match the result even if only its prefix matches the tree content (same effect as adding "/**" to every tree path)
copy (bool) – If True, make copy of the supplied data; otherwise, just make it the root.

Warning

If copy==False, the root data is already assumed to be normalized. If it isn’t, the behavior might be incorrect.

copy()[source]¶: Get a full copy the prefix tree

match(path)[source]¶: Return the match result for the path

add_entry(path, value, force=False, branch_option='normalize')¶

Add value to a given path (overwrite leaf value if necessary).

Doesn’t replace leaves with branches and vice-verse if force==False.

Parameters:

path –
value –
force (bool) – If True, change leaf into a branch and vice-versa; otherwise, raises ValueError if the conversion is necessary.
branch_option (str) –
Decides what to do if the value is dictionary-like:
- 'attach' – just attach the root,
- 'copy' – copy and attach,
- 'normalize' – copy while normalizing all the keys according to the current rules.

as_dict(style='nested', copy=True)¶

Convert into a dict object.

Parameters:	style (str) – Determines style of the result: `'nested'` – subtrees are turned into nested dictionaries, `'flat'` – single dictionary is formed with full paths as keys. copy (bool) – If `False` and `style=='nested'`, return the root dictionary.

static as_dictionary(obj, case_normalization=None)¶

Convert object into Dictionary with the given parameters.

If object is already a Dictionary (or its subclass), return unchanged, even if its parameters are different.

as_json(style='nested')¶

Convert into a JSON string.

Parameters:	style (str) – Determines style of the result: - `'nested'` – subtrees are turned into nested dictionaries, - `'flat'` – single dictionary is formed with full paths as keys.

as_pandas(index_key=True, as_series=True)¶

Convert into a pandas DataFrame or Series object.

Parameters:	index_key (bool) – If `False`, create a 2-column table with the first column (`"key"`) containing string path and the second column (`"value"`) containing value; otherwise, move key to the table index. as_series (bool) – If `index_key==True` and `as_series==True`, convert the resulting DataFrame into 1D Series (the key is the index); otherwise, keep it as a single-column table

asdict(style='nested', copy=True)¶

Convert into a dict object.

Parameters:	style (str) – Determines style of the result: `'nested'` – subtrees are turned into nested dictionaries, `'flat'` – single dictionary is formed with full paths as keys. copy (bool) – If `False` and `style=='nested'`, return the root dictionary.

branch_copy(branch='')¶: Get a copy of the branch as a Dictionary

branch_pointer(branch='')¶: Get a DictionaryPointer of a given branch

collect(paths, detach=False, ignore_missing=True)¶

Collect a set of subpaths into a separate dictionary.

Parameters:	paths – list or set of paths detach – if `True`, added branches are removed from this dictionary ignore_missing – if `True`, ignore paths from the list which are not present in this dictionary; otherwise, raise a `KeyError`.

del_entry(path)¶

Delete entry from the dictionary.

Return True if the path was present. Note that it never raises KeyError.

detach(path)¶

Remove a branch or a leaf from the current dictionary.

Branch is returned as a separate Dictionary. If path is missing, raise a KeyError.

diff(other)¶

Perform an element-wise comparison to another Dictionary.

If the other Dictionary has a different case sensitivity, raise ValueError.

Returns:	`DictionaryDiff`

static diff_flatdict(first, second)¶

Find the difference between flat dict objects.

Returns:	`DictionaryDiff`

filter_self(pred, to_visit='leafs', pass_path=False, topdown=False)¶

Remove all the nodes from the dictionary for which pred returns False.

Parameters:

pred (callable) – Filter function. Leafs are passed to pred by value, branches (if visited) are passed as DictionaryPointer.
to_visit (str) – Can be 'leafs', 'branches' or 'all' and determines which parts of the dictionary passed to the predicate.
pass_path (bool) – If True, pass the node path (in the form of a normalized list) as a first argument to pred.
topdown (bool) – If True, visit node and its leafs before its subtrees leafs.

static find_intersection(dicts, use_flatten=False)¶

Find intersection of multiple dictionaries.

Parameters:	dicts ([`Dictionary`]) – use_flatten (bool) – If `True` flatten all dictionaries before comparison (works faster for a large number of dictionaries).
Returns:	`DictionaryIntersection`

classmethod from_json(data, case_normalization=None)¶: Convert JSON representations of a dictionary into a Dictionary object

get(path, default=None)¶: Analog of dict.get(): D.get(k,d) -> D[k] if k in D else d

get_entry(path, as_pointer=False)¶

Get entry at a given path

Parameters:	path – as_pointer (bool) – If `True` and entry is not a leaf, return `DictionaryPointer`; otherwise, return `Dictionary`

get_matching_paths(pattern, wildkey='*', wildpath='**', only_leaves=True)¶

Get all paths in the tree that match the provided pattern.

Parameters:	wildkey (str) – Pattern symbol that matches any key. wildpath (str) – Pattern symbol that matches any subpath (possibly empty). only_leaves (bool) – If `True`, only check leaf paths; otherwise, check subtree paths (i.e., incomplete leaf paths) as well. Basically, `only_leaves=False` is analogous to adding wildpath at the end of the pattern.

get_matching_subtree(pattern, wildkey='*', wildpath='**', only_leaves=True)¶

Get a subtree containing nodes with paths matching the provided pattern.

Parameters:	wildkey (str) – Pattern symbol that matches any key. wildpath (str) – Pattern symbol that matches any subpath (possibly empty). only_leaves (bool) – If `True`, only check leaf paths; otherwise, check subtree paths (i.e., incomplete leaf paths) as well. Basically, `only_leaves=False` is analogous to adding wildpath at the end of the pattern.

get_max_prefix(path, kind='all')¶

Find the longest prefix of path contained in the dictionary.

Return tuple (prefix, rest), where both path entries are normalized according to the dictionary rules (i.e., these are lists representing normalized paths). kind determines which kind of path to consider and can be 'leaf', 'branch' or 'all'. If the longest prefix is of a different kind, return (None,None).

get_path()¶

has_entry(path, kind='all')¶

Determine if the path is in the dictionary.

kind determines which kind of path to consider and can be 'leaf', 'branch' or 'all'.

is_branch_path(path)¶: Determine if the path is in the dictionary and points to a branch

static is_dictionary(obj, generic=True)¶

Determine if the object is a dictionary.

Parameters:	obj – generic (bool) – if False, passes only `Dictionary` (or subclasses) objects; otherwise, passes any dictionary-like object.
Returns:	bool

is_leaf_path(path)¶: Determine if the path is in the dictionary and points to a leaf

items(ordered=False, leafs=False, path_kind='split', wrap_branches=True)¶

Analog of dict.items(), by default iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

iteritems(ordered=False, leafs=False, path_kind='split', wrap_branches=True)¶

Analog of dict.items(), by default iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

iterkeys(ordered=False, leafs=False, path_kind='split')¶

Analog of dict.keys(), iterating only over the immediate children of the root.

Parameters:	ordered (bool) – If `True`, loop over keys in alphabetic order. leafs (bool) – If `True`, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary) path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

iternodes(to_visit='leafs', ordered=False, include_path=False, topdown=False)¶

Iterate over nodes.

Parameters:	to_visit (str) – Can be `'leafs'`, `'branches'` or `'all'` and determines which parts of the dictionary are visited. ordered (bool) – If `True`, loop over paths in alphabetic order. include_path (bool) – Include in the return value. topdown (bool) – If `True`, visit node and its leafs before its subtrees leafs.
Yields:	Values for leafs and `DictionaryPointer` for branches. If `include_path==True`, yields tuple `(path, value)`, where path is in the form of a normalized list.

itervalues(ordered=False, leafs=False, wrap_branches=True)¶

Analog of dict.values(), iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

keys(ordered=False, leafs=False, path_kind='split')¶

Analog of dict.keys(), iterating only over the immediate children of the root.

Parameters:	ordered (bool) – If `True`, loop over keys in alphabetic order. leafs (bool) – If `True`, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary) path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

map_self(func, to_visit='leafs', pass_path=False, topdown=False, branch_option='normalize')¶

Apply func to the nodes in the dictionary.

Note that any pointers to the replaced branches or their sub-branches will become invalid.

Parameters:

func (callable) – Mapping function. Leafs are passed by value, branches (if visited) are passed as DictionaryPointer.
to_visit (str) – Can be 'leafs', 'branches' or 'all' and determines which parts of the dictionary passed to the map function.
pass_path (bool) – If True, pass the node path (in the form of a normalized list) as a first argument to func.
topdown (bool) – If True, visit node and its leafs before its subtrees leafs.
branch_option (str) – If the function returns a dict-like object, determines how to incorporate into the dictionary; can be "normalize" (make a copy with normalized paths and insert that), "copy" (make a copy without normalization), or "attach" (simply replace the value without copying and normalization)

merge(source, path='', overwrite=True, normalize_paths=True)¶

Attach source (dict or other Dictionary) to a given branch; source is automatically deep-copied.

If source is not a dictionary, simply assign it (i.e., D.merge(v,p) is equivalent to D.add_entry(p,v,force=True) in this case). Compared to add_entry(), merges two branches instead of removing the old branch completely.

Parameters:	source (dict or Dictionary) – branch (tuple or str) – Destination path. overwrite (bool) – If `True`, replaces the old entries with the new ones (it only matters for leaf assignments). normalize_paths (bool) – If `True` and the dictionary isn’t case sensitive, perform normalization if the source.

nodes(to_visit='leafs', ordered=False, include_path=False, topdown=False)¶

Iterate over nodes.

Parameters:	to_visit (str) – Can be `'leafs'`, `'branches'` or `'all'` and determines which parts of the dictionary are visited. ordered (bool) – If `True`, loop over paths in alphabetic order. include_path (bool) – Include in the return value. topdown (bool) – If `True`, visit node and its leafs before its subtrees leafs.
Yields:	Values for leafs and `DictionaryPointer` for branches. If `include_path==True`, yields tuple `(path, value)`, where path is in the form of a normalized list.

paths(ordered=False, topdown=False, path_kind='split')¶

Return list of all paths (leafs and nodes).

Parameters:	ordered (bool) – If `True`, loop over paths in alphabetic order. topdown (bool) – If `True`, return node’s leafs before its subtrees leafs. path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

pop(path, default=None)¶

Analog of dict.pop(): remove value at path and return it if path in D, otherwise return default

Note that it never raises KeyError.

setdefault(path, default=None)¶: Analog of dict.setdefault(): D.setdefault(k,d) -> D.get(k,d), also sets D[k]=d if k not in D.

size()¶: Return the total size of the dictionary (number of nodes)

update(source, path='', overwrite=True, normalize_paths=True)¶

Attach source (dict or other Dictionary) to a given branch; source is automatically deep-copied.

If source is not a dictionary, simply assign it (i.e., D.merge(v,p) is equivalent to D.add_entry(p,v,force=True) in this case). Compared to add_entry(), merges two branches instead of removing the old branch completely.

Parameters:	source (dict or Dictionary) – branch (tuple or str) – Destination path. overwrite (bool) – If `True`, replaces the old entries with the new ones (it only matters for leaf assignments). normalize_paths (bool) – If `True` and the dictionary isn’t case sensitive, perform normalization if the source.

updated(source, path='', overwrite=True, normalize_paths=True)¶

Get a copy of the dictionary and attach a new branch to it.

Parameters are the same as in the Dictionary.merge().

values(ordered=False, leafs=False, wrap_branches=True)¶

Analog of dict.values(), iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

viewitems(ordered=False, leafs=False, path_kind='split', wrap_branches=True)¶

Analog of dict.items(), by default iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
path_kind (str) – either "split" (each path is a tuple of individual keys), or "joined" (each path is a single string)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

viewkeys(ordered=False, leafs=False, path_kind='split')¶

Analog of dict.keys(), iterating only over the immediate children of the root.

Parameters:	ordered (bool) – If `True`, loop over keys in alphabetic order. leafs (bool) – If `True`, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary) path_kind (str) – either `"split"` (each path is a tuple of individual keys), or `"joined"` (each path is a single string)

viewvalues(ordered=False, leafs=False, wrap_branches=True)¶

Analog of dict.values(), iterating only over the immediate children of the root.

Parameters:

ordered (bool) – If True, loop over keys in alphabetic order.
leafs (bool) – If True, loop over leaf nodes (i.e., behave as ‘flat’ dictionary); otherwise, loop over immediate children (i.e., behave as ‘nested’ dictionary)
wrap_branches (bool) – if True, wrap sub-branches into DictionaryPointer objects; otherwise, return them as nested built-in dictionaries

class pylablib.core.utils.dictionary.PrefixShortcutTree(shortcuts=None)[source]¶

Bases: object

Convenient storage for dictionary path shortcuts.

Parameters:	shortcuts (dict) – Dictionary of shortcuts `{shortcut: full_path}`.

copy()[source]¶: Return full copy

add_shortcut(source, dest, exact=False)[source]¶

Add a single shortcut.

Parameters:	source – Shortcut path. dest – expanded path corresponding to the shortcut. exact (bool) – If `True`, the shortcut works only for the exact path; otherwise, it works for any path with ‘source’ as a prefix.

add_shortcuts(shortcuts, exact=False)[source]¶

Add a dictionary of shortcuts {shortcut: full_path}.

Arguments are the same as in PrefixShortcutTree.add_shortcut().

remove_shortcut(source)[source]¶: Remove a shortcut from the tree

updated(shortcuts, exact=False)[source]¶

Make a copy and add additional shortcuts.

Arguments are the same as in PrefixShortcutTree.add_shortcuts().

class pylablib.core.utils.dictionary.DictionaryNode(**vargs)[source]¶: Bases: object

pylablib.core.utils.dictionary.dict_to_object_local(data, name=None, object_generator=<function _default_object_generator>)[source]¶

class pylablib.core.utils.dictionary.ItemAccessor(getter=None, setter=None, deleter=None, contains_checker='auto', normalize_names=True, path_separator=None, missing_error=None)[source]¶

Bases: object

Simple wrapper which implements array interface using supplied methods.

Also has an option to normalize requested paths (enabled by default)

Parameters:

getter – method for getting values (None means none is supplied, so getting raises an error)
setter – method for setting values (None means none is supplied, so setting raises an error)
deleter – method for deleting values (None means none is supplied, so deleting raises an error)
contains_checker – method for checking if variable is present (None means none is supplied, so checking containment raises an error; "auto" means that getter raising KeyError is used for checking)
normalize_names – if True, normalize a supplied path using the standard Dictionary rules and join it into a single string using the supplied separator
path_separator – path separator regex used for splitting and joining the supplied paths (by default, the standard "/" separator)
missing_error – if not None, specifies the error raised on the missing value; used in __contains__, get() and setdefault() to determine if the value is missing

get(name, default=None)[source]¶

setdefault(name, default=None)[source]¶

pylablib.core.utils.files module¶

Utilities for working with the file system: creating/removing/listing folders, comparing folders and files, working with zip archives.

pylablib.core.utils.files.eof(f, strict=False)[source]¶

Standard EOF function.

Return True if the the marker is at the end of the file. If strict==True, only return True if the marker is exactly at the end of file; otherwise, return True if it’s at the end of further.

pylablib.core.utils.files.get_file_creation_time(path, timestamp=True)[source]¶

Try to find a file creation time. Return current time if an error occurs.

If timestamp==True, return UNIX timestamp; otherwise, return datetime.datetime.

pylablib.core.utils.files.get_file_modification_time(path, timestamp=True)[source]¶

Try to find a file modification time. Return current time if an error occurs.

If timestamp==True, return UNIX timestamp; otherwise, return datetime.datetime

pylablib.core.utils.files.touch(fname, times=None)[source]¶

Update file access and modification times.

Parameters:	times (tuple) – Access and modification times; if times is `None`, use current time.

pylablib.core.utils.files.generate_indexed_filename(name_format, idx_start=0, folder='')[source]¶

Generate an unused indexed filename in folder.

The name has name_format (using standard Python format() rules, e.g., "data_{:03d}.dat"), and the index starts with idx_start.

pylablib.core.utils.files.generate_prefixed_filename(prefix='', suffix='', idx_start=None, idx_fmt='d', folder=None)[source]¶

Generate an unused filename with the given prefix and suffix in the given folder.

By default, the format is prefix_{:d}_suffix, where the parameter is the index starting with idx_start. If idx_start is None, first check simply prefix+suffix name before using numbered indices.

pylablib.core.utils.files.generate_temp_filename(prefix='__tmp__', idx_start=0, idx_template='d', folder='')[source]¶

Generate a temporary filename with a given prefix.

idx_template is the number index format (only the parameter itself, not the whole string).

pylablib.core.utils.files.fullsplit(path, ignore_empty=True)[source]¶

Split path into a list.

If ignore_empty==True, exclude empty folder names.

pylablib.core.utils.files.normalize_path(p)[source]¶: Normalize filesystem path (case and origin). If two paths are identical, they should be equal when normalized

pylablib.core.utils.files.case_sensitive_path()[source]¶: Check if OS path names are case-sensitive (e.g., Linux)

pylablib.core.utils.files.paths_equal(a, b)[source]¶: Determine if the two paths are equal (can be local or have different case)

pylablib.core.utils.files.relative_path(a, b, check_paths=True)[source]¶

Determine return path a as seen from b.

If check_paths==True, check if a is contained in b and raise the OSError if it isn’t.

pylablib.core.utils.files.is_path_valid(p)[source]¶

Check if the string is a valid path.

Not guaranteed to have complete success rate, but catches most likely errors (invalid characters, reserved file names, too long, etc.) Does not check if the path actually exists or if it can be written into.

class pylablib.core.utils.files.TempFile(folder='', name=None, mode='w', wait_time=None, rep_time=None)[source]¶

Bases: object

Temporary file context manager.

Upon creation, generate an unused temporary filename. Upon entry, create the file using supplied mode and return self. Upon exit, close and remove the file.

Can be mostly substituted by tempfile.TemporaryFile(), but generates file locally, and with specified/determined name. Preserved largely for legacy reasons.

Parameters:	folder (str) – Containing folder. name (str) – File name. If `None`, generate new temporary name. mode (str) – File opening mode. wait_time (float) – Waiting time between attempts to create the file if the first try fails. rep_time (int) – Number of attempts to create the file if the first try fails.

f¶: File object.

name¶

File name.

Type:	str

full_name¶

File name including containing folder.

Type:	str

pylablib.core.utils.files.copy_file(source, dest, overwrite=True, cmp_on_overwrite=True, preserve_metadata=True)[source]¶

Copy file, creating a containing folder if necessary. Return True if the operation was performed.

Parameters:	overwrite (bool) – If `True`, overwrite existing file. cmp_on_overwrite (bool) – If `True` and the two files are compared to be the same, don’t perform overwrite. preserve_metadata (bool) – If `True`, preserve file metadata (such as creation time) by using `shutil.copy2()`; otherwise, use `shutil.copy()`

pylablib.core.utils.files.move_file(source, dest, overwrite=True, cmp_on_overwrite=True, preserve_if_not_move=False)[source]¶

Move file, creating a containing folder if necessary. Returns True if the operation was performed.

Parameters:	overwrite (bool) – If `True`, overwrite existing file (if the existing file isn’t overwritten, preserve the original). cmp_on_overwrite (bool) – If `True` and the two files are compared to be the same, don’t perform overwrite. preserve_if_not_move (bool) – If `True` and the files are identical, preserve the original.

pylablib.core.utils.files.ensure_dir_singlelevel(path, error_on_file=True)[source]¶

pylablib.core.utils.files.ensure_dir(path, error_on_file=True)[source]¶

Ensure that the folder exists (create a new one if necessary).

If error_on_file==True, raise OSError if there’s a file with the same name.

pylablib.core.utils.files.remove_dir(path, error_on_file=True)[source]¶

Remove the folder recursively if it exists.

If error_on_file==True, raise OSError if there’s a file with the same name.

pylablib.core.utils.files.remove_dir_if_empty(path, error_on_file=True)[source]¶

Remove the folder only if it’s empty.

If error_on_file==True, raise OSError if there’s a file with the same name.

pylablib.core.utils.files.clean_dir(path, error_on_file=True)[source]¶

Remove the folder and then recreate it.

If error_on_file==True, raise OSError if there’s a file with the same name.

class pylablib.core.utils.files.FolderList[source]¶

Bases: pylablib.core.utils.files.FolderList

Describes folder content

count()¶: Return number of occurrences of value.

files¶

folders¶

index()¶

Return first index of value.

Raises ValueError if the value is not present.

pylablib.core.utils.files.list_dir(folder='', folder_filter=None, file_filter=None, separate_kinds=True, error_on_file=True)[source]¶

Return folder content filtered by folder_filter and file_filter.

Parameters:

folder (str) – Path to the folder.
folder_filter – Folder filter function (more description at string.get_string_filter()).
file_filter – File filter function (more description at string.get_string_filter()).
separate_kinds (bool) – if True, return FolderList with files and folder separate; otherwise, return a single list (works much faster).
error_on_file (bool) – if True, raise OSError if there’s a file with the same name as the target folder.

pylablib.core.utils.files.dir_empty(folder, folder_filter=None, file_filter=None, level='single', error_on_file=True)[source]¶

Check if the folder is empty (only checks content filtered by folder_filter and file_filter).

Parameters:

folder (str) – Path to the folder.
folder_filter – Folder filter function (more description at string.get_string_filter()).
file_filter – File filter function (more description at string.get_string_filter()).
level (str) – if 'single', check only immediate folder content; if 'recursive', follow recursively in all folders passing folder_filter.
error_on_file (bool) – if True, raise OSError if there’s a file with the same name as the target folder.

pylablib.core.utils.files.walk_dir(folder, folder_filter=None, file_filter=None, rel_path=True, topdown=True, visit_folder_filter=None, max_depth=None)[source]¶

Modification of os.walk() function.

Acts in a similar way, but followlinks is always False and errors of os.listdir() are always passed.

Parameters:

folder (str) – Path to the folder.
folder_filter – Folder filter function (more description at string.get_string_filter()).
file_filter – File filter function (more description at string.get_string_filter()).
rel_path (bool) – If True, the returned folder path is specified relative to the initial path.
topdown (bool) – If True, return folder before its subfolders.
visit_folder_filter – Filter for visiting folders (more description at string.get_string_filter()). If not None, specifies filter for visiting folders which is different from folder_filter (filter for returned folders).
max_depth (int) – If not None, limits the recursion depth.

Yields:

For each folder (including the original) yields a tuple (folder_path, folders, files),: where folder_path is the containing folder name and folders and files are its content (similar to list_dir()).

pylablib.core.utils.files.list_dir_recursive(folder, folder_filter=None, file_filter=None, topdown=True, visit_folder_filter=None, max_depth=None)[source]¶

Recursive walk analog of list_dir().

Parameters are the same as walk_dir().

Returns:	`FolderList`

pylablib.core.utils.files.copy_dir(source, dest, folder_filter=None, file_filter=None, overwrite=True, cmp_on_overwrite=True, preserve_metadata=True)[source]¶

Copy files satisfying the filtering conditions.

Parameters:

source (str) – Source path.
dest (str) – Destination path.
folder_filter – Folder filter function (more description at string.get_string_filter()).
file_filter – File filter function (more description at string.get_string_filter()).
overwrite (bool) – If True, overwrite existing files.
cmp_on_overwrite (bool) – If True and the two files are compared to be the same, don’t perform overwrite.
preserve_metadata (bool) – If True, preserve file metadata (such as creation time) by using shutil.copy2(); otherwise, use shutil.copy()

pylablib.core.utils.files.move_dir(source, dest, folder_filter=None, file_filter=None, overwrite=True, cmp_on_overwrite=True, preserve_if_not_move=False)[source]¶

Move files satisfying the filtering conditions.

Parameters:

source (str) – Source path.
dest (str) – Destination path.
folder_filter – Folder filter function (more description at string.get_string_filter()).
file_filter – File filter function (more description at string.get_string_filter()).
overwrite (bool) – If True, overwrite existing files (if the existing file isn’t overwritten, preserve the original).
cmp_on_overwrite (bool) – If True and the two files are compared to be the same, don’t perform overwrite.
preserve_if_not_move (bool) – If True and the files are identical, preserve the original.

pylablib.core.utils.files.combine_diff(d1, d2)[source]¶

pylablib.core.utils.files.cmp_dirs(a, b, folder_filter=None, file_filter=None, shallow=True, return_difference=False)[source]¶

Compare the folders based on the content filtered by folder_filter and file_filter.

Parameters:

a (str) – First folder path
b (str) – Second folder path
folder_filter – Folder filter function (more description at string.get_string_filter()).
file_filter – File filter function (more description at string.get_string_filter()).
shallow – If True, do shallow comparison of the files (see filecmp.cmp()).
return_difference – If False, simply return bool; otherwise, return difference type ('=', '+', '-' or '*').

pylablib.core.utils.files.retry_copy(source, dest, overwrite=True, cmp_on_overwrite=True, preserve_metadata=True, try_times=5, delay=0.3)[source]¶

Retrying version of copy_file().

If the operation raises error, wait for delay (in seconds) and call it again. Try total of try_times times.

pylablib.core.utils.files.retry_move(source, dest, overwrite=True, cmp_on_overwrite=True, preserve_if_not_move=False, try_times=5, delay=0.3)[source]¶: Retrying version of move_file() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_remove(path, try_times=5, delay=0.3)[source]¶: Retrying version of os.remove() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_ensure_dir(path, error_on_file=True, try_times=5, delay=0.3)[source]¶: Retrying version of ensure_dir() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_copy_dir(source, dest, folder_filter=None, file_filter=None, overwrite=True, cmp_on_overwrite=True, preserve_metadata=True, try_times=5, delay=0.3)[source]¶: Retrying version of copy_dir() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_move_dir(source, dest, folder_filter=None, file_filter=None, overwrite=True, cmp_on_overwrite=True, preserve_if_not_move=False, try_times=5, delay=0.3)[source]¶: Retrying version of move_dir() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_remove_dir(path, error_on_file=True, try_times=5, delay=0.3)[source]¶: Retrying version of remove_dir() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_remove_dir_if_empty(path, error_on_file=True, try_times=5, delay=0.3)[source]¶: Retrying version of remove_dir_if_empty() (see retry_copy() for details on retrying).

pylablib.core.utils.files.retry_clean_dir(path, error_on_file=True, try_times=5, delay=0.3)[source]¶: Retrying version of clean_dir() (see retry_copy() for details on retrying).

pylablib.core.utils.files.zip_folder(zip_path, source_path, inside_path='', folder_filter=None, file_filter=None, mode='a', compression=8, compresslevel=None)[source]¶

Add a folder into a zip archive.

Parameters:

zip_path (str) – Path to the .zip file.
source_path (str) – Path to the source folder.
inside_path (str) – Destination path inside the zip archive.
folder_filter – Folder filter function (more description at string.get_string_filter()).
file_filter – File filter function (more description at string.get_string_filter()).
mode (str) – Zip archive adding mode (see zipfile.ZipFile).
compression – Zip archive compression (see zipfile.ZipFile).
compresslevel – Zip archive compression level (see zipfile.ZipFile); ignored for Python version below 3.7.

pylablib.core.utils.files.zip_file(zip_path, source_path, inside_name=None, mode='a', compression=8, compresslevel=None)[source]¶

Add a file into a zip archive.

Parameters:

zip_path (str) – Path to the .zip file.
source_path (str) – Path to the source file.
inside_name (str) – Destination file name inside the zip archive (source name on the top level by default).
mode (str) – Zip archive adding mode (see zipfile.ZipFile).
compression – Zip archive compression (see zipfile.ZipFile).
compresslevel – Zip archive compression level (see zipfile.ZipFile); ignored for Python version below 3.7.

pylablib.core.utils.files.zip_multiple_files(zip_path, source_paths, inside_names=None, mode='a', compression=8, compresslevel=None)[source]¶

Add a multiple files into a zip archive.

Parameters:

zip_path (str) – Path to the .zip file.
source_paths ([str]) – List of path to the source files.
inside_names ([str] or None) – List of destination file names inside the zip archive (source name on the top level by default).
mode (str) – Zip archive adding mode (see zipfile.ZipFile).
compression – Zip archive compression (see zipfile.ZipFile).
compresslevel – Zip archive compression level (see zipfile.ZipFile); ignored for Python version below 3.7.

pylablib.core.utils.files.unzip_folder(zip_path, dest_path, inside_path='', folder_filter=None, file_filter=None)[source]¶

Extract a folder from a zip archive (create containing folder if necessary).

Parameters:

zip_path (str) – Path to the .zip file.
dest_path (str) – Path to the destination folder.
inside_path (str) – Source path inside the zip archive; extracted data paths are relative (i.e., they don’t include inside_path).
folder_filter – Folder filter function (more description at string.get_string_filter()).
file_filter – File filter function (more description at string.get_string_filter()).

pylablib.core.utils.files.unzip_file(zip_path, dest_path, inside_path)[source]¶

Extract a file from a zip archive (create containing folder if necessary).

Parameters:	zip_path (str) – Path to the .zip file. dest_path (str) – Destination file path. inside_path (str) – Source path inside the zip archive.

pylablib.core.utils.funcargparse module¶

Contains routines for checking arguments passed into a function for better flexibility.

pylablib.core.utils.funcargparse.parameter_value_error(par_val, par_name, message=None, error_type=None)[source]¶: Raise parameter value error (ValueError by default).

pylablib.core.utils.funcargparse.parameter_range_error(par_val, par_name, par_set=None, message=None, error_type=None)[source]¶: Raise parameter range error (ValueError by default).

pylablib.core.utils.funcargparse.check_parameter_range(par_val, par_name, par_set, message=None, error_type=None)[source]¶: Raise error if par_val is not in in the par_set (par_name is used in the error message).

pylablib.core.utils.funcargparse.getdefault(value, default_value, unassigned_value=None, conflict_action='ignore', message=None, error_type=None)[source]¶

Analog of dict’s getdefault.

If value is unassigned_value, return default_value instead. If conflict_action=='error' and value!=default_value, raise value error using message and error_type.

pylablib.core.utils.funcargparse.is_sequence(value, sequence_type='builtin;nostring')[source]¶

Check if value is a sequence.

sequence_type is semicolon separated list of possible sequence types:

'builtin' - list, tuple or str
'nostring' - str is not allows
'array' - list, tuple or numpy.ndarray
'indexable' - anything which can be indexed
'haslength' - anything with length property

pylablib.core.utils.funcargparse.make_sequence(element, length=1, sequence_type='list')[source]¶: Turn element into a sequence of sequence_type ('list' or 'tuple') repeated length times.

pylablib.core.utils.funcargparse.as_sequence(value, multiply_length=1, allowed_type='builtin;nostring', wrapping_type='list', length_conflict_action='ignore', message=None, error_type=None)[source]¶

Ensure that value is a sequence.

If value is not a sequence of allowed_type (as checked by is_sequence()), turn it into a sequence specified by wrapping_type and multiply_length.

If value is a sequence and length_conflict_action=='error', raise error with error_type and error_message if the length doesn’t match multiply_length. Otherwise, return value unchanged.

pylablib.core.utils.functions module¶

Utilities for dealing with function, methods and function signatures.

class pylablib.core.utils.functions.FunctionSignature(arg_names=None, defaults=None, varg_name=None, kwarg_name=None, kwonly_arg_names=None, cls=None, obj=None, name=None, doc=None)[source]¶

Bases: object

Description of a function signature, including name, argument names, default values, names of varg and kwarg arguments, class and object (for methods) and docstring.

Parameters:

arg_names (list) – Names of the arguments.
default (dict) – Dictionary {name: value} of default values.
varg_name (str) – Name of *varg parameter (None means no such parameter).
kwarg_name (str) – Name of **kwarg parameter (None means no such parameter).
cls – Caller class, for methods.
obj – Caller object, for methods.
name (str) – Function name.
doc (str) – Function docstring.

get_defaults_list()[source]¶: Get list of default values for arguments in the order specified in the signature.

signature(pass_order=None)[source]¶

Get string containing a signature (arguments list) of the function (call or definition), including *vargs and **kwargs.

If pass_order is not None, it specifies the order in which the arguments are passed.

wrap_function(func, pass_order=None)[source]¶

Wrap a function func into a containing function with this signature.

Sets function name, argument names, default values, object and class (for methods) and docstring. If pass_order is not None, it determines the order in which the positional arguments are passed to the wrapped function.

as_kwargs(args, kwargs, add_defaults=False, exclude=None)[source]¶

Turn args and kwargs into a single kwargs dictionary using the names of positional arguments.

If add_defaults==True, add all the non-specified default arguments as well. If the function takes *args argument and some of the supplied arguments go there, place them into a list under "*" key in the result. If exclude is not None is specifies arguments which should be excluded.

arg_value(argname, args=None, kwargs=None)[source]¶: Get the value of the argument with the given name for given args and kwargs

mandatory_args_num()[source]¶

Get minimal number of arguments which have to be passed to the function.

The mandatory arguments are the ones which are not bound to caller object (i.e., not self) and don’t have default values.

max_args_num(include_positional=True, include_keywords=True)[source]¶

Get maximal number of arguments which can be passed to the function.

Parameters:	include_positional (bool) – If `True` and function accepts `vargs`, return `None` (unlimited number of arguments). include_keywords* (bool) – If `True` and function accepts `**kwargs`, return `None` (unlimited number of arguments).

static from_function(func, follow_wrapped=True)[source]¶

Get signature of the given function or method.

If follow_wrapped==True, follow __wrapped__ attributes until the innermost function (useful for getting signatures of functions wrapped using functools methods).

copy()[source]¶: Return a copy

as_simple_func()[source]¶

Turn the signature into a simple function (as opposed to a bound method).

If the signature corresponds to a bound method, get rid of the first argument in the signature (self) and the bound object. Otherwise, return unchanged.

static merge(inner, outer, add_place='front', merge_duplicates=True, overwrite=None, hide_outer_obj=False)[source]¶

Merge two signatures (used for wrapping functions).

The signature describes the function would take arguments according to the outer signature and pass them according to the inner signature.

The arguments are combined:

if add_place=='front', the outer arguments are placed in the beginning, followed by inner arguments not already listed;
if add_place=='back', the inner arguments are placed in the beginning, followed by outer arguments not already listed.

The default values are joined, with the outer values superseding the inner values.

overwrite is a set or a list specifying which inner parameters are overwritten by the outer. It includes 'name', 'doc', 'cls', 'obj', 'varg_name' and 'kwarg_name'; the default value is all parameters.

If the inner signature is a bound method and hide_inner_obj==True, treat it as a function (with self argument missing). In this case, the wrapped signature .obj field will be None.

Returns:	`(signature, pass_order)` pass_order is the order in which the arguments of the combined signature may be passed to the inner signature; it may be different from the signature order if `add_place=='front'`. If `merge_duplicates==True`, duplicate entries in pass_order are omitted; otherwise, they’re repeated.
Return type:	tuple

pylablib.core.utils.functions.funcsig(func, follow_wrapped=True)[source]¶: Return a function signature object

pylablib.core.utils.functions.getargsfrom(source, **merge_params)[source]¶

Decorator factory.

Returns decorator that conforms function signature to the source function. **merge_params are passed to the FunctionSignature.merge() method merging wrapped and source signature.

The default behavior (conforming parameter names, default values args and kwargs names) is useful for wrapping universal functions like g(*args, **kwargs).

Example:

def f(x, y=2):
    return x+y

@getargsfrom(f)
def g(*args): # Now g has the same signature as f, including parameter names and default values.
    return prod(args)

pylablib.core.utils.functions.call_cut_args(func, *args, **kwargs)[source]¶: Call func with the given arguments, omitting the ones that don’t fit its signature.

pylablib.core.utils.functions.getattr_call(obj, attr_name, *args, **vargs)[source]¶

Call the getter for the attribute attr_name of obj.

If the attribute is a property, pass *args and **kwargs to the getter (fget); otherwise, ignore them.

pylablib.core.utils.functions.setattr_call(obj, attr_name, *args, **vargs)[source]¶

Call the setter for the attribute attr_name of obj.

If the attribute is a property, pass *args and **kwargs to the setter (fset); otherwise, the set value is assumed to be either the first argument, or the keyword argument with the name 'value'.

pylablib.core.utils.functions.delattr_call(obj, attr_name, *args, **vargs)[source]¶

Call the deleter for the attribute attr_name of obj.

If the attribute is a property, pass *args and **kwargs to the deleter (fdel); otherwise, ignore them.

class pylablib.core.utils.functions.IObjectCall[source]¶

Bases: object

Universal interface for object method call (makes methods, attributes and properties look like methods).

Should be called with an object as a first argument.

class pylablib.core.utils.functions.MethodObjectCall(method)[source]¶

Bases: pylablib.core.utils.functions.IObjectCall

Object call created from an object method.

Parameters:	method – Either a method object or a method name which is used for the call.

class pylablib.core.utils.functions.AttrObjectCall(name, as_getter)[source]¶

Bases: pylablib.core.utils.functions.IObjectCall

Object call created from an object attribute (makes attributes and properties look like methods).

Parameters:	name (str) – Attribute name. as_getter (bool) – If `True`, call the getter when invoked; otherwise, call the setter.

If an attribute is a simple attribute, than getter gets no arguments and setter gets one argument (either the first argument, or the keyword argument named 'value'). If it’s a property, pass all the parameters to the property call.

class pylablib.core.utils.functions.IObjectProperty[source]¶

Bases: object

Universal interface for an object property (makes methods, attributes and properties look like properties).

Can be used to get, set or remove a property.

get(obj, params=None)[source]¶

set(obj, value)[source]¶

rem(obj, params=None)[source]¶

class pylablib.core.utils.functions.MethodObjectProperty(getter=None, setter=None, remover=None, expand_tuple=True)[source]¶

Bases: pylablib.core.utils.functions.IObjectProperty

Object property created from object methods (makes methods look like properties).

Parameters:

getter (callable) – Method invoked on get(). If None, raise RuntimeError when called.
setter (callable) – Method invoked on set(). If None, raise RuntimeError when called.
remover (callable) – Method invoked on rem(). If None, raise RuntimeError when called.
expand_tuple (bool) – If True and if the first argument in the method call is a tuple, expand it as an argument list for the underlying function call.

get(obj, params=None)[source]¶

set(obj, value)[source]¶

rem(obj, params=None)[source]¶

class pylablib.core.utils.functions.AttrObjectProperty(name, use_getter=True, use_setter=True, use_remover=True, expand_tuple=True)[source]¶

Bases: pylablib.core.utils.functions.IObjectProperty

Object property created from object attribute. Works with attributes or properties.

Parameters:

name (str) – Attribute name.
use_getter (bool) – If False, raise RuntimeError when calling get method.
use_setter (bool) – If False, raise RuntimeError when calling set method.
use_remover (bool) – If False, raise RuntimeError when calling rem method.
expand_tuple (bool) – If True and if the first argument in the method call is a tuple, expand it as an argument list for the underlying function call.

get(obj, params=None)[source]¶

set(obj, value)[source]¶

rem(obj, params=None)[source]¶

pylablib.core.utils.functions.empty_object_property(value=None)[source]¶: Dummy property which does nothing and returns value on get (None by default).

pylablib.core.utils.functions.obj_prop(*args, **kwargs)[source]¶

Build an object property wrapper.

If no arguments (or a single None argument) are supplied, return a dummy property. If one argument is supplied, return AttrObjectProperty for a property with a given name. Otherwise, return MethodObjectProperty property.

pylablib.core.utils.functions.as_obj_prop(value)[source]¶

Turn value into an object property using obj_prop() function.

If it’s already IObjectProperty, return unchanged. If value is a tuple, expand as an argument list.

pylablib.core.utils.functions.delaydef(gen)[source]¶

Wrapper for a delayed definition of a function inside of a module.

Useful if defining a function is computationally costly. The wrapped function should be a generator of the target function rather than the function itself.

On the first call the generator is executed to define the target function, which is then substituted for all subsequent calls.

pylablib.core.utils.general module¶

Collection of small utilities.

pylablib.core.utils.general.set_props(obj, prop_names, props)[source]¶

Set multiple attributes of obj.

Names are given by prop_names list and values are given by props list.

pylablib.core.utils.general.get_props(obj, prop_names)[source]¶

Get multiple attributes of obj.

Names are given by prop_names list.

pylablib.core.utils.general.getattr_multivar(obj, attrs, **kwargs)[source]¶

Try to get an attribute of obj given a list attrs of its potential names.

If no attributes are found and default keyword argument is supplied, return this default value; otherwise, raise AttributeError.

pylablib.core.utils.general.using_method(func, method_name=None, inherit_signature=True)[source]¶

Decorator that makes the function attempt to call the first argument’s method instead of func.

Before calling the function, try and call a method of the first argument named method_name (func name by default). If the method exists, call it instead of the wrapped function. If inherit_signature==True, completely copy the signature of the wrapped method (name, args list, docstring, etc.).

pylablib.core.utils.general.to_predicate(x)[source]¶

Turn x into a predicate.

If x is callable, it will be called with a single argument and returned value determines if the argument passes. If x is a container, an argument passes if it’s contained in x.

pylablib.core.utils.general.map_container(value, func)[source]¶

Map values in the container.

value can be a tuple, a list or a dict (mapping is applied to the values) raises ValueError if it’s something else.

pylablib.core.utils.general.as_container(val, t)[source]¶

Turn iterable into a container of type t.

Can handle named tuples, which have different constructor signature.

pylablib.core.utils.general.recursive_map(value, func)[source]¶

Map container recursively.

value can be a tuple, a list or a dict (mapping is applied to the values).

pylablib.core.utils.general.make_flat_namedtuple(nt, fields=None, name=None, subfield_fmt='{field:}_{subfield:}')[source]¶

Turn a nested structure of named tuples into a single flat namedtuple.

Parameters:

nt – toplevel namedtuple class to be flattened
fields – a dictionary {name: desc} of the fields, where name is the named tuple name, and desc is either a nested namedtuple class, or a list of arguments which are passed to the recursive call to this function (e.g., [TTuple, {"field": TNestedTuple}]). Any tuple field which is present in this dictionary gets recursively flattened, and the field names of the corresponding returned tuple are added to the full list of fields
name – name of the resulting tuple
subfield_fmt – format string, which describes how the combined field name is built out of the original field name and the subtuple field name; by default, connect with "_", i.e., t.field.subfiled turns into t.field_subfield.

Returns:

a new namedtuple class, which describes the flattened structure

pylablib.core.utils.general.any_item(d)[source]¶: Return arbitrary tuple (key, value) contained in the dictionary (works both in Python 2 and 3)

pylablib.core.utils.general.merge_dicts(*dicts)[source]¶

Combine multiple dict objects together.

If multiple dictionaries have the same keys, later arguments have higher priority.

pylablib.core.utils.general.filter_dict(pred, d, exclude=False)[source]¶

Filter dictionary based on a predicate.

pred can be a callable or a container (in which case the predicate is true if a value is in the container). If exclude==True, the predicate is inverted.

pylablib.core.utils.general.map_dict_keys(func, d)[source]¶: Map dictionary keys with func

pylablib.core.utils.general.map_dict_values(func, d)[source]¶: Map dictionary values with func

pylablib.core.utils.general.to_dict(d, default=None)[source]¶

Convert a dict or a list of pairs or single keys (or mixed) into a dict.

If a list element is single, default value is used.

pylablib.core.utils.general.to_pairs_list(d, default=None)[source]¶

Convert a dict or a list of pairs or single keys (or mixed) into a list of pairs.

If a list element is single, default value is used. When converting list into list, the order is preserved.

pylablib.core.utils.general.invert_dict(d, kmap=None)[source]¶

Invert dictionary (switch keys and values).

If kmap is supplied, it’s a function mapping dictionary values into inverted dictionary keys (identity by default).

pylablib.core.utils.general.flatten_list(l)[source]¶: Flatten nested list/tuple structure into a single list.

pylablib.core.utils.general.partition_list(pred, l)[source]¶: Split the lis` l into two parts based on the predicate.

pylablib.core.utils.general.split_in_groups(key_func, l, continuous=True, max_group_size=None)[source]¶

Split the list l into groups according to the key_func.

Go over the list and group the elements with the same key value together. If continuous==False, groups all elements with the same key together regardless of where they are in the list. otherwise, group only continuous sequences of the elements with the same key together (element with different key in the middle will result in two groups). If continuous==True and max_group_size is not None, it determines the maximal size of a group; larger groups are split into separate groups.

pylablib.core.utils.general.sort_set_by_list(s, l, keep_duplicates=True)[source]¶

Convert the set s into a list ordered by a list l.

Elements in s which are not in l are omitted. If keep_duplicates==True, keep duplicate occurrences in l in the result; otherwise, only keep the first occurrence.

pylablib.core.utils.general.compare_lists(l1, l2, sort_lists=False, keep_duplicates=True)[source]¶

Return three lists (l1 and l2, l1-l2, l2-l1).

If sort_lists==True, sort the first two lists by l1, and the last one by l2; otherwise, the order is undefined. If sort_lists==True, keep_duplicated determines if duplicate elements show up in the result.

pylablib.core.utils.general.topological_order(graph, visit_order=None)[source]¶

Get a topological order of a graph.

Return a list of nodes where each node is listed after its children. If visit_order is not None, it is a list specifying nodes visiting order (nodes earlier in the list are visited first). Otherwise, the visit order is undefined. graph is a dictionary {node: [children]}. If graph contains loops, raise ValueError.

class pylablib.core.utils.general.DummyResource[source]¶

Bases: object

Object that acts as a resource (has __enter__ and __exit__ methods), but doesn’t do anything.

Analog of:

@contextlib.contextmanager
def dummy_resource():
    yield

class pylablib.core.utils.general.RetryOnException(tries=None, exceptions=None)[source]¶

Bases: object

Wrapper for repeating the same block of code several time if an exception occurs

Useful for filesystem or communication operations, where retrying a failed operation is a valid option.

Parameters:	tries (int) – Determines how many time will the chunk of code execute before re-raising the exception; `None` (default) means no limit exceptions (Exception or list) – A single exception class or a `list` of exception classes which are going to be silenced.

Example:

for t in RetryOnException(tries,exceptions):
    with t:
        ... do stuff ...

is analogue of:

for i in range(tries):
    try:
        ... do stuff ...
    except exceptions:
        if i==tries-1:
            raise

class ExceptionCatcher(retrier, try_number)[source]¶

Bases: object

reraise()[source]¶

pylablib.core.utils.general.retry_wait(func, try_times=1, delay=0.0, exceptions=None)[source]¶

Try calling function (with no arguments) at most try_times as long as it keeps raising exception.

If exceptions is not None, it specifies which exception types should be silenced. If an exception has been raised, wait delay seconds before retrying.

class pylablib.core.utils.general.SilenceException(exceptions=None, on_exception=None, reraise=False)[source]¶

Bases: object

Context which silences exceptions raised in a block of code.

Parameters:	exceptions (Exception or list) – A single exception class or a list of exception classes which are going to be silenced. on_exception (callable) – A callback to be invoked if an exception occurs. reraise (bool) – Defines if the exception is re-raised after the callback has been invoked.

A simple bit of syntax sugar. The code:

with SilenceException(exceptions,on_exception,reraise):
    ... do stuff ...

is exactly analogous to:

try:
    ... do stuff ...
except exceptions:
    on_exception()
    if reraise:
        raise

pylablib.core.utils.general.full_exit(code=<Signals.SIGTERM: 15>)[source]¶

Terminate the current process and all of its threads.

Doesn’t perform any cleanup or resource release; should only be used if the process is irrevocably damaged.

class pylablib.core.utils.general.UIDGenerator(thread_safe=False)[source]¶

Bases: object

Generator of unique numeric IDs.

Parameters:	thread_safe (bool) – If `True`, using lock to ensure that simultaneous calls from different threads are handled properly.

reset(value=0)[source]¶: Reset the generator to the given value

class pylablib.core.utils.general.NamedUIDGenerator(name_template='{0}{1:03d}', thread_safe=False)[source]¶

Bases: object

Generator of unique string IDs based on a name.

Parameters:	name_template (str) – Format string with two parameters (name and numeric ID) used to generate string IDs. thread_safe (bool) – If `True`, using lock to ensure that simultaneous calls from different threads are handled properly.

pylablib.core.utils.general.call_limit(func, period=1, cooldown=0.0, limit=None, default=None)[source]¶

Wrap func such that calls to it are forwarded only under certain conditions.

If period>1, then func is called after at least period calls to the wrapped function. If cooldown>0, then func is called after at least cooldown seconds passed since the last call. if limit is not None, then func is called only first limit times. If several conditions are specified, they should be satisfied simultaneously. default specifies return value if func wasn’t called. Returned function also has an added method reset, which resets the internal call and time counters.

pylablib.core.utils.general.doc_inherit(parent)[source]¶

Wrapper for inheriting docstrings from parent classes.

Takes parent class as an argument and replaces the docstring of the wrapped function by the docstring of the same-named function from the parent class (if available).

class pylablib.core.utils.general.Countdown(timeout, start=True)[source]¶

Bases: object

Object for convenient handling of timeouts and countdowns with interrupts.

Parameters:	timeout (float) – Countdown timeout; if `None`, assumed to be infinite. start (bool) – if `True`, automatically start the countdown; otherwise, wait until `trigger()` is called explicitly

reset(start=True)[source]¶: Restart the countdown from the current moment

trigger(restart=True)[source]¶

Trigger the countdown.

If restart==True, restart the countdown if it’s running; otherwise, do nothing in that situation.

running()[source]¶: Check if the countdown is running

stop()[source]¶: Stop the timer if currently running

time_left(t=None, bound_below=True)[source]¶

Return the amount of time left. For infinite timeout, return None.

If bound_below==True, instead of negative time return zero. If t is supplied, it indicates the current time; otherwise, use time.time().

add_time(dt, t=None, bound_below=True)[source]¶

Add a given amount of time (positive or negative) to the start time (timeout stays the same).

If bound_below==True, do not let the end time (start time plus timeout) to get below the current time. If t is supplied, it indicates the current time; otherwise, use time.time().

set_timeout(timeout)[source]¶: Change the timer timeout

time_passed()[source]¶: Return the amount of time passed since the countdown start/reset, or None if it is not started

passed()[source]¶: Check if the timeout has passed

class pylablib.core.utils.general.Timer(period, skip_first=False)[source]¶

Bases: object

Object for keeping time of repeating tasks.

Parameters:	period (float) – Timer period.

change_period(period, method='current')[source]¶

Change the timer period.

method specifies the changing method. Could be "current" (change the period of the ongoing tick), "next" (change the period starting from the next tick), "reset_skip" (reset the timer and skip the first tick) or "reset_noskip" (reset the timer and don’t skip the first tick).

reset(skip_first=False)[source]¶

Reset the timer.

If skip_first==False, timer ticks immediately; otherwise, it starts ticking only after one period.

time_left(t=None, bound_below=True)[source]¶

Return the amount of time left before the next tick.

If bound_below==True, instead of negative time return zero.

passed(t=None)[source]¶

Return the number of ticks passed.

If timer period is zero, always return 1.

acknowledge(n=None, nmin=0)[source]¶

Acknowledge the timer tick.

n specifies the number of tick to acknowledge (by default, all passed). Return number of actually acknowledged ticks (0 if the timer hasn’t ticked since the last acknowledgement).

class pylablib.core.utils.general.StreamFileLogger(path, stream=None, lock=None, autoflush=False)[source]¶

Bases: object

Stream logger that replaces standard output stream (usually stdout or stderr) and logs them into a file.

Parameters:

path – path to the destination logfile. The file is always appended.
stream – an optional output stream into which the output will be duplicated; usually, the original stream which is being replaced
lock – a thread lock object, which is used for any file writing operation; necessary if replacing standard streams (such as sys.stdout or sys.stderr) in a multithreading environment.
autoflush – if True, flush after any write operation into stream

It is also possible to subclass the file and overload write_header() method to write a header before the first file write operation during the execution.

The intended use is to log stdout or stderr streams:

import sys, threading
sys.stderr = StreamFileLogger("error_log.txt", stream=sys.stderr, lock=threading.Lock())

write_header(f)[source]¶: Write header to file stream f

add_path(path)[source]¶: Add another logging path to the list

remove_path(path)[source]¶: Remove logging path to the list

write(s)[source]¶

flush()[source]¶

pylablib.core.utils.general.setbp()¶

pylablib.core.utils.general.timing(n=1, name=None, profile=False)[source]¶

Context manager for timing a piece of code.

Measures the time it takes to execute the wrapped code and prints the result.

Parameters:	n – can specify the number of repetitions, which is used to show time per single repetition. name – name which is printed alongside the time profile – if `True`, use `cProfile` and print its output instead of a simple timing

class pylablib.core.utils.general.AccessIterator(obj, access_function=None)[source]¶

Bases: object

Simple sequential access iterator with customizable access function (by default it’s 1D indexing).

Determines end of iterations by IndexError.

Parameters:	obj – Container to be iterated over. access_function (callable) – A function which takes two parameters obj and idx and either returns the element or raises `IndexError`. By default, a simple `__getitem__` operation.

next()¶

pylablib.core.utils.general.muxcall(argname, special_args=None, mux_argnames=None, return_kind='list', allow_partial=False)[source]¶

Wrap a function such that it can become multiplexable over a given argument.

Parameters:

argname – name of the argument to loop over
special_args – if not None, defines a dictionary {arg: func} for special values of the argument (e.g., "all", None, etc.), where arg is its value, and func is the method taking the same arguments as the called function and returning the substitute argument (e.g., a list of all arguments)
mux_argnames – names of additional arguments which, when supplied list or dict values, and when the argname value is a list, specify different values for different calls
return_kind – method to combined multiple returned values; can be "list", "dict" (return dict {arg: result}), or "none" (simply return None)
allow_partial – if True and some of mux_argnames argument do not specify value for the full range of argname value, do not call the function for those unspecified values; otherwise (allow_partial is True), the error will be raised

pylablib.core.utils.general.wait_for_keypress(message='Waiting...')[source]¶

pylablib.core.utils.general.restart()[source]¶

Restart the script.

Execution will not resume after this call. Note: due to Windows limitations, this function does not replace the current process with a new one, but rather calls a new process and makes the current one wait for its execution. Hence, each nested call adds an additional loaded application into the memory. Therefore, nesting restart calls (i.e., calling several restarts in a row) should be avoided.

pylablib.core.utils.indexing module¶

Processing and normalization of different indexing styles.

pylablib.core.utils.indexing.string_list_idx(names_to_find, names_list, only_exact=False)[source]¶

Index through a list of strings in names_list.

Return corresponding numerical indices. Case sensitive; first look for exact matching, then for prefix matching (unless only_exact=True).

pylablib.core.utils.indexing.is_slice(idx)[source]¶: Check if idx is slice.

pylablib.core.utils.indexing.is_range(idx)[source]¶: Check if idx is iterable (list, numpy array, or builtins.range).

pylablib.core.utils.indexing.is_bool_array(idx)[source]¶: Check if idx is a boolean array.

pylablib.core.utils.indexing.to_range(idx, length)[source]¶: Turn list, array, builtins.range, slice into an iterable.

pylablib.core.utils.indexing.covers_all(idx, length, strict=False, ordered=True)[source]¶

Check if idx covers all of the elements (indices from 0 to length).

If strict==True, strictly checks the condition; otherwise may return False even if idx actually covers everything, but takes less time (i.e., can be used for optimization). If ordered==True, only returns True when indices follow in order.

class pylablib.core.utils.indexing.IIndex[source]¶

Bases: object

A generic index object.

Used to transform a variety of indexes into a subset applicable for specific objects (numpy arrays or lists).

Allowed input index types:

scalar: integer, string
vector: integer lists or numpy arrays, bool lists or numpy arrays, string lists or numpy arrays, builtin.ranges, slices and string slices

tup()[source]¶: Represent index as a tuple for easy unpacking.

class pylablib.core.utils.indexing.NumpyIndex(idx, ndim=None)[source]¶

Bases: pylablib.core.utils.indexing.IIndex

NumPy compatible index: allows for integers, slices, numpy integer or boolean arrays, integer lists or builtin.ranges.

Parameters:	idx – raw index ndim – index dimensionality (either 0 or 1); if supplied, assume that idx is already normalized

tup()¶: Represent index as a tuple for easy unpacking.

class pylablib.core.utils.indexing.ListIndex(idx, names=None, ndim=None)[source]¶

Bases: pylablib.core.utils.indexing.IIndex

List compatible index: allows for integers, slices, numpy integer arrays, integer lists or builtin.ranges.

Parameters:	idx – raw index names – list of allowed index string values, which is used to convert them into integers ndim – index dimensionality (either 0 or 1); if supplied, assume that idx is already normalized

tup()¶: Represent index as a tuple for easy unpacking.

class pylablib.core.utils.indexing.ListIndexNoSlice(idx, names=None, length=None, ndim=None)[source]¶

Bases: pylablib.core.utils.indexing.ListIndex

List compatible index with slice unwrapped into builtin.range: allows for integers, numpy integer arrays, integer lists or builtin.ranges.

Parameters:	idx – raw index names – list of allowed index string values, which is used to convert them into integers length – length of the list (used to expand slice indices) ndim – index dimensionality (either 0 or 1); if supplied, assume that idx is already normalized

tup()¶: Represent index as a tuple for easy unpacking.

pylablib.core.utils.indexing.to_double_index(idx, names)[source]¶

Convert double index into a pair of indexes.

Assume that one index is purely numerical, while the other can take names (out of the supplied list).

Parameters:	idx – raw double index names – list of allowed index string values, which is used to convert them into integers

pylablib.core.utils.ipc module¶

Universal interface for inter-process communication.

Focus on higher throughput for large numpy arrays via shared memory.

class pylablib.core.utils.ipc.IIPCChannel[source]¶

Bases: object

Generic IPC channel interface

send(data)[source]¶: Send data

recv(timeout=None)[source]¶: Receive data

send_numpy(data)[source]¶: Send numpy array

recv_numpy(timeout=None)[source]¶: Receive numpy array

get_peer_args()[source]¶: Get arguments required to create a peer connection

classmethod from_args(*args)[source]¶: Create a peer connection from the supplied arguments

class pylablib.core.utils.ipc.TPipeMsg(id, data)¶

Bases: tuple

count()¶: Return number of occurrences of value.

data¶

id¶

index()¶

Return first index of value.

Raises ValueError if the value is not present.

class pylablib.core.utils.ipc.PipeIPCChannel(pipe_conn=None)[source]¶

Bases: pylablib.core.utils.ipc.IIPCChannel

Generic IPC channel interface using pipe.

get_peer_args()[source]¶: Get arguments required to create a peer connection

send(data)[source]¶: Send data

recv(timeout=None)[source]¶: Receive data

classmethod from_args(*args)¶: Create a peer connection from the supplied arguments

recv_numpy(timeout=None)¶: Receive numpy array

send_numpy(data)¶: Send numpy array

class pylablib.core.utils.ipc.SharedMemIPCChannel(pipe_conn=None, arr=None, arr_size=None)[source]¶

Bases: pylablib.core.utils.ipc.PipeIPCChannel

Generic IPC channel interface using pipe and shared memory for large arrays.

get_peer_args()[source]¶: Get arguments required to create a peer connection

send_numpy(data, method='auto', timeout=None)[source]¶: Send numpy array

recv_numpy(timeout=None)[source]¶: Receive numpy array

classmethod from_args(*args)¶: Create a peer connection from the supplied arguments

recv(timeout=None)¶: Receive data

send(data)¶: Send data

class pylablib.core.utils.ipc.TShmemVarDesc(offset, size, kind, fixed_size)¶

Bases: tuple

count()¶: Return number of occurrences of value.

fixed_size¶

index()¶

Return first index of value.

Raises ValueError if the value is not present.

kind¶

offset¶

size¶

class pylablib.core.utils.ipc.SharedMemIPCTable(pipe_conn=None, arr=None, arr_size=None, lock=True)[source]¶

Bases: object

Shared memory table for exchanging shared variables between processes.

Can be used instead of channels for variables which are rarely changed but frequently checked (e.g., status), or when synchronization of sending and receiving might be difficult

add_variable(name, size, kind='pickle')[source]¶

Add a variable with a given name.

The variable info is also communicated to the other endpoint. size determines maximal variable size in bytes. If the actual size ever exceeds it, an exception will be raised. kind determines the way to convert variable into bytes; can be "pickle" (universal, but large size overhead), "nps_###"` (where ### can be any numpy scalar dtype description, e.g., "float" or "<u2") for numpy scalars, or "npa_###"` (where ### means the same as for nps) for numpy arrays (in this case the array size and shape need to be communicated separately).

set_variable(name, value)[source]¶

Set a variable with a given name.

If the variable is missing, raise an exception.

get_variable(name, default=None)[source]¶

Get a variable with a given name.

If the variable is missing, return default.

is_peer_connected()[source]¶: Check if the peer is connected (i.e., the other side of the pipe is initialized)

close_connection()[source]¶: Mark the connection as closed

is_peer_closed()[source]¶: Check if the peer is closed

get_peer_args()[source]¶: Get arguments required to create a peer connection

classmethod from_args(*args)[source]¶: Create a peer connection from the supplied arguments

pylablib.core.utils.library_parameters module¶

Storage for global library parameters

pylablib.core.utils.library_parameters.temp_library_parameters(restore=None)[source]¶

Context manager, which restores library parameters upon exit.

If rester is not None, it can specify a list of parameters to be restored (by default, all parameters).

pylablib.core.utils.module module¶

Library for dealing with python module properties.

pylablib.core.utils.module.get_package_version(pkg)[source]¶

Get the version of the package.

If the package version is unavailable, return None.

pylablib.core.utils.module.cmp_versions(ver1, ver2)[source]¶

Compare two package versions.

Return '<' if the first version is older (smaller), '>' if it’s younger (larger) or '=' if it’s the same.

pylablib.core.utils.module.cmp_package_version(pkg, ver)[source]¶

Compare current package version to ver.

ver should be a name of the package (rather than the module). Return '<' if current version is older (smaller), '>' if it’s younger (larger) or '=' if it’s the same. If the package version is unavailable, return None.

pylablib.core.utils.module.expand_relative_path(module_name, rel_path)[source]¶

Turn a relative module path into an absolute one.

module_name is the absolute name of the reference module, rel_path is the path relative to this module.

pylablib.core.utils.module.get_loaded_package_modules(pkg_name)[source]¶

Get all modules in the package pkg_name.

Returns a dict {name: module}.

pylablib.core.utils.module.get_imported_modules(module, explicit=False)[source]¶

Get modules imported within a given module.

If explicit==True, take into account only toplevel objects which are modules (corresponds to import module or from package import module statements) If explicit==False, also include all modules containing toplevel objects (corresponds to from module import Class or from package import function statements). Return a dictionary {name: module} (modules with the same name are considered to be the same).

pylablib.core.utils.module.get_reload_order(modules)[source]¶

Find reload order for modules which respects dependencies (a module is loaded before its dependents).

modules is a dict {name: module}.

The module dependencies (i.e., the modules which the current module depends on) are determined based on imported modules and modules containing toplevel module objects.

pylablib.core.utils.module.reload_package_modules(pkg_name, ignore_errors=False)[source]¶

Reload package pkg_name, while respecting dependencies of its submodules.

If ignore_errors=True, ignore ImportError exceptions during the reloading process.

pylablib.core.utils.module.unload_package_modules(pkg_name, ignore_errors=False)[source]¶

Reload package pkg_name, while respecting dependencies of its submodules.

If ignore_errors=True, ignore ImportError exceptions during the reloading process.

pylablib.core.utils.module.get_library_path()[source]¶: Get a filesystem path for the pyLabLib library (the one containing current the module).

pylablib.core.utils.module.get_library_name()[source]¶: Get the name for the pyLabLib library (the one containing current the module).

pylablib.core.utils.module.get_executable(console=False)[source]¶

Get Python executable.

If console==True and the current executable is windowed (i.e., "pythonw.exe"), return the corresponding "python.exe" instead.

pylablib.core.utils.module.get_python_folder()[source]¶: Return Python interpreter folder (the folder containing the python executable)

pylablib.core.utils.module.pip_install(pkg, upgrade=False)[source]¶

Call pip install for a given package.

If upgrade==True, call with --upgrade key (upgrade current version if it is already installed).

pylablib.core.utils.module.install_if_older(pkg, min_ver='')[source]¶

Install pkg from the default PyPI repository if its version is lower that min_ver

If min_ver is None, upgrade to the newest version regardless; if min_ver=="", install only if no version is installed. Return True if the package was installed.

pylablib.core.utils.nbtools module¶

pylablib.core.utils.nbtools.c_array[source]¶: Generate a numba C-ordered array type with the given element type, number of dimensions, and read-only flag

pylablib.core.utils.nbtools.au1(x, off)[source]¶: Extract a little-endian 1-byte unsigned integer from a numpy byte array at the given offset

pylablib.core.utils.nbtools.au2(x, off)[source]¶: Extract a little-endian 2-byte unsigned integer from a numpy byte array at the given offset

pylablib.core.utils.nbtools.au4(x, off)[source]¶: Extract a little-endian 4-byte unsigned integer from a numpy byte array at the given offset

pylablib.core.utils.nbtools.au8(x, off)[source]¶: Extract a little-endian 8-byte unsigned integer from a numpy byte array at the given offset

pylablib.core.utils.nbtools.ai1(x, off)[source]¶: Extract a little-endian 1-byte unsigned integer from a numpy byte array at the given offset

pylablib.core.utils.nbtools.ai2(x, off)[source]¶: Extract a little-endian 2-byte unsigned integer from a numpy byte array at the given offset

pylablib.core.utils.nbtools.ai4(x, off)[source]¶: Extract a little-endian 4-byte unsigned integer from a numpy byte array at the given offset

pylablib.core.utils.nbtools.ai8(x, off)[source]¶: Extract a little-endian 8-byte unsigned integer from a numpy byte array at the given offset

pylablib.core.utils.nbtools.copy_array_chunks[source]¶

Generate and compile a numba function for copying an array in chunks. base specifies the base array type (by default, unsigned byte); if par==True, generate a parallelized implementation. if nogil==True, use the nogil numba option to release GIL during the execution.

The returned function takes 4 arguments: source array, destination array, number of chunks, and size (in elements) of each chunk.

pylablib.core.utils.nbtools.copy_array_strided[source]¶

Generate and compile a numba function for copying an array in chunks with an arbitrary stride. base specifies the base array type (by default, unsigned byte); if par==True, generate a parallelized implementation. if nogil==True, use the nogil numba option to release GIL during the execution.

The returned function takes 6 arguments: source array, destination array, number of chunks, size (in elements) of each chunk, chunks stride (in elements) in the source array, and offset (in elements) from the beginning of the first array. If size is the same as stride and the offset is zero, this function would mimic the one generated by copy_array_chunks().

pylablib.core.utils.net module¶

A wrapper for built-in TCP/IP routines.

exception pylablib.core.utils.net.SocketError[source]¶

Bases: OSError

Base socket error class.

args¶

characters_written¶

errno¶: POSIX exception code

filename¶: exception filename

filename2¶: second exception filename

strerror¶: exception strerror

with_traceback()¶: Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception pylablib.core.utils.net.SocketTimeout[source]¶

Bases: pylablib.core.utils.net.SocketError

Socket timeout error.

args¶

characters_written¶

errno¶: POSIX exception code

filename¶: exception filename

filename2¶: second exception filename

strerror¶: exception strerror

with_traceback()¶: Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

pylablib.core.utils.net.get_local_addr()[source]¶: Get local IP address

pylablib.core.utils.net.get_all_local_addr()[source]¶: Get a list of all local IP addresses

pylablib.core.utils.net.get_local_hostname(full=True)[source]¶: Get a local host name

pylablib.core.utils.net.get_all_remote_addr(hostname)[source]¶: Get a list of all remote addresses of a remote host by name

pylablib.core.utils.net.get_remote_hostname(addr, error_on_missing=False)[source]¶: Get a remote host name by its address

pylablib.core.utils.net.as_addr_port(addr, port)[source]¶

Parse the given address and port combination.

addr can be a host address, a tuple (addr, port), or a string "addr:port"; in the first case the given port is used, while in the other two it is ignore. Return tuple (addr, port).

class pylablib.core.utils.net.ClientSocket(sock=None, timeout=None, wait_callback=None, send_method='decllen', recv_method='decllen', datatype='auto', nodelay=False)[source]¶

Bases: object

A client socket (used to connect to a server socket).

Parameters:

sock (socket.socket) – Socket to wrap; if None create a new one.
timeout (float) – The timeout used for connecting and sending/receiving (None means no timeout).
wait_callback (callable) – Called periodically (every 100ms by default) while waiting for connecting or sending/receiving.
send_method (str) – Default sending method.
recv_method (str) – Default receiving method.
datatype (str) – Type of the returned data; can be "bytes" (return bytes object), "str" (return str object), or "auto" (default Python result: str in Python 2 and bytes in Python 3)
nodelay (bool) – Whether to enable TCP_NODELAY.

Possible sending/receiving methods are:

'fixedlen': data is sent as is, and receiving requires to know the length of the message;
'decllen': data is prepended by a length, and receiving reads this length and doesn’t need predetermined length info.

sock¶

Corresponding Python socket.

Type:	socket.socket

decllen_bo¶

Byteorder of the prepended length for 'decllen' sending method. Can be either '>' (big-endian, default) or '<'.

Type:	str

decllen_ll¶

Length of the prepended length for 'decllen' sending method; default is 4 bytes (corresponding to maximum of 4Gb per single length-prepended message)

Type:	int

set_wait_callback(wait_callback=None)[source]¶: Set callback function for waiting during connecting or sending/receiving

set_timeout(timeout=None)[source]¶: Set timeout for connecting or sending/receiving

get_timeout()[source]¶: Get timeout for connecting or sending/receiving

using_timeout(timeout=None)[source]¶: Context manager for usage of a different timeout inside a block

connect(host, port)[source]¶: Connect to a remote host

close()[source]¶: Close the connection

is_connected()[source]¶: Check if the connection is opened

get_local_name()[source]¶: Return IP address and port of this socket

get_peer_name()[source]¶: Return IP address and port of the peer socket

recv_fixedlen(l)[source]¶: Receive fixed-length message of length l

recv_delimiter(delim, lmax=None, chunk_l=1024, strict=False)[source]¶

Receive a single message ending with a delimiter delim (can be several characters, or list several possible delimiter strings).

lmax specifies the maximal received length (None means no limit). chunk_l specifies the size of data chunk to be read in one try. If strict==False, keep receiving as much data as possible until a delimiter is found in the end (only works properly if a single line is expected); otherwise, receive the data byte-by-byte and stop as soon as a delimiter is found (equivalent to setting chunk_l=1).

recv_decllen()[source]¶

Receive variable-length message (prepended by its length).

Length format is described by decllen_bo and decllen_ll attributes.

recv(l=None)[source]¶: Receive a message using the default method.

recv_all(chunk_l=1024)[source]¶

Receive all of the data currently in the socket.

chunk_l specifies the size of data chunk to be read in one try. For technical reasons, use 1ms timeout (i.e., this operation takes 1ms).

recv_ack(l=None)[source]¶: Receive a message using the default method and send an acknowledgement (message length)

send_fixedlen(msg)[source]¶: Send a message as is

send_decllen(msg)[source]¶

Send a message as a variable-length (prepending its length in the sent message).

Length format is described by decllen_bo and decllen_ll attributes.

send_delimiter(msg, delimiter)[source]¶: Send a message with a delimiter delim (can be several characters)

send(msg)[source]¶: Send a message using the default method.

send_ack(msg)[source]¶

Send a message using default method and wait for acknowledgement (message length).

If the acknowledgement message length doesn’t agree, raise SocketError.

pylablib.core.utils.net.recv_JSON(sock, chunk_l=1024, strict=True)[source]¶

Receive a complete JSON token from the socket.

chunk_l specifies the size of data chunk to be read in one try. If strict==False, keep receiving as much data as possible until the received data forms a complete JSON token. otherwise, receive the data byte-by-byte and stop as soon as a token is formed (equivalent to setting chunk_l=1).

pylablib.core.utils.net.listen(host, port, conn_func, port_func=None, wait_callback=None, timeout=None, backlog=10, wrap_socket=True, connections_number=None, socket_kwargs=None)[source]¶

Run a server socket at the given host and port.

Parameters:

host (str) – Server host address. If None, use the local host defined by socket.gethostname().
port (int) – Server port. If 0, generate an arbitrary free port.
conn_func (callable) – Called with the client socket as a single argument every time a connection is established.
port_func (callable) – Called with the port as a single argument when the listening starts (useful with port=0).
wait_callback (callable) – A callback function which is called periodically (every 100ms by default) while awaiting for connections.
timeout (float) – Timeout for waiting for the connections (None is no timeout).
backlog (int) – Backlog length for the socket (see socket.socket.listen()).
wrap_socket (bool) – If True, wrap the client socket of the connection into ClientSocket class; otherwise, return socket.socket object.
connections_number (int) – Specifies maximal number of connections before the listening function returns (by default, the number is unlimited).
socket_kwargs (dict) – additional keyword arguments passed to ClientSocket constructor.

Checking for connections is paused until conn_func returns. If multiple simultaneous connections are expected, conn_func should spawn a separate processing thread and return. If connections_number is None (i.e., there’s no limit on the number of connections before closing), this function never returns.

pylablib.core.utils.numerical module¶

Numerical functions that don’t deal with sequences.

pylablib.core.utils.numerical.gcd(*numbers)[source]¶: Euclid’s algorithm for GCD. Arguments are cast to integer

pylablib.core.utils.numerical.integer_distance(x)[source]¶: Get distance to the closes integer

pylablib.core.utils.numerical.gcd_approx(a, b, min_fraction=1e-08, tolerance=1e-05)[source]¶

Approximate Euclid’s algorithm for possible non-integer values.

Try to find a number d such that a/d and b/d are less than tolerance away from a closest integer. If GCD becomes less than min_fraction * min(a, b), raise ArithmeticError.

pylablib.core.utils.numerical.round_significant(x, n)[source]¶: Rounds x to n significant digits (not the same as n decimal places!).

pylablib.core.utils.numerical.limit_to_range(x, min_val=None, max_val=None, default=0)[source]¶

Confine x to the given limit.

Default limit values are None, which means no limit. default specifies returned value if both x, min_val and max_val are None.

class pylablib.core.utils.numerical.infinite_list(start=0, step=1)[source]¶

Bases: object

Mimics the behavior of the usual list, but is infinite and immutable.

Supports accessing elements, slicing (including slices giving infinite lists) and iterating. Iterating over it naturally leads to an infinite loop, so it should only be used either for finite slices or for loops with break condition.

Parameters:	start – The first element of the list. step – List step.

class counter(lst)[source]¶

Bases: object

next()¶

pylablib.core.utils.numerical.unity()[source]¶: Return a unity function

pylablib.core.utils.numerical.constant(c)[source]¶

Return a function which returns a constant c.

c can only be either a scalar, or an array-like object with the shape matching the expected argument.

pylablib.core.utils.numerical.polynomial(coeffs)[source]¶

Return a polynomial function which with coefficients coeffs.

Coefficients are list lowest-order first, so that coeffs[i] is the coefficient in front of x**i.

pylablib.core.utils.observer_pool module¶

A simple observer pool (notification pool) implementation.

class pylablib.core.utils.observer_pool.ObserverPool(expand_tuple=True)[source]¶

Bases: object

An observer pool.

Stores notification functions (callbacks), and calls them whenever notify() is called. The callbacks can have priority (higher priority ones are called first) and filter (observer is only called if the filter function passes the notification tag).

Parameters:	expand_tuple (bool) – if `True` and the notification value is a tuple, treat it as an argument list for the callback functions.

class Observer(filt, callback, priority, attr, cacheable)¶

Bases: tuple

attr¶

cacheable¶

callback¶

count()¶: Return number of occurrences of value.

filt¶

index()¶

Return first index of value.

Raises ValueError if the value is not present.

priority¶

add_observer(callback, name=None, filt=None, priority=0, attr=None, cacheable=False)[source]¶

Add the observer callback.

Parameters:

callback (callable) – callback function; takes at least one argument (notification tag), and possible more depending on the notification value.
name (str) – stored callback name; by default, a unique name is auto-generated
filt (callable or None) – a filter function for this observer (the observer is called only if the notify() function tag and value pass the filter); by default, all tags are accepted
priority (int) – callback priority; higher priority callback are invoked first.
attr – additional observer attributes (can be used by ObserverPool subclasses to change their behavior).
cacheable (bool) – if True, assumes that the filter function only depends on the tag, so its calls can be cached.

Returns:

callback name (equal to name if supplied; an automatically generated name otherwise).

remove_observer(name)[source]¶: Remove the observer callback with the given name

find_observers(tag, value)[source]¶

notify(tag, value=())[source]¶

Notify the observers by calling their callbacks.

Return a dictionary of the callback results. By default the value is an empty tuple: for expand_tuple==True this means that only one argument (tag) is passed to the callbacks.

pylablib.core.utils.py3 module¶

Dealing with Python2 / Python3 compatibility.

pylablib.core.utils.py3.as_str(data)[source]¶: Convert a string into a text string

pylablib.core.utils.py3.as_bytes(data)[source]¶: Convert a string into bytes

pylablib.core.utils.py3.as_builtin_bytes(data)[source]¶: Convert a string into bytes

pylablib.core.utils.py3.as_datatype(data, datatype)[source]¶

Convert a string into a given datatypes.

datatype can be "str" (text string), "bytes" (byte string), or "auto" (no conversion).

pylablib.core.utils.rpyc_utils module¶

Routines and classes related to RPyC package

pylablib.core.utils.rpyc_utils.obtain(proxy, serv=None, deep=False, direct=False)[source]¶

Obtain a remote netref object by value (i.e., copy it to the local Python instance).

Wrapper around rpyc.utils.classic.obtain() with some special cases handling. serv specifies the current remote service. If it is of type SocketTunnelService, use its socket tunnel for faster transfer. If deep==True and proxy is a container (tuple, list, or dict), run the function recursively for all its sub-elements. If direct==True, directly use RPyC obtain method; otherwise use the custom method, which works better with large numpy arrays, but worse with composite types (e.g., lists).

pylablib.core.utils.rpyc_utils.transfer(obj, serv)[source]¶

Send a local object to the remote PC by value (i.e., copy it to the remote Python instance).

A ‘reversed’ version of obtain().

class pylablib.core.utils.rpyc_utils.SocketTunnelService(server=False)[source]¶

Bases: sphinx.ext.autodoc.importer._MockObject

Extension of the standard rpyc.core.service.SlaveService with built-in network socket tunnel for faster data transfer.

In order for the tunnel to work, services on both ends need to be subclasses of SocketTunnelService. Because of the initial setup protocol, the two services are asymmetric: one should be ‘server’ (corresponding to the listening server), and one should be ‘client’ (external connection). The roles are decided by the server constructor parameter.

tunnel_send(obj, packer=None)[source]¶

Send data through the socket tunnel.

If packer is not None, it defines a function to convert obj to a bytes string.

tunnel_recv(unpacker=None)[source]¶

Receive data sent through the socket tunnel.

If unpacker is not None, it defines a function to convert the received bytes string into an object.

obtain(proxy)[source]¶: Execute obtain() on the local instance

transfer(obj)[source]¶: Execute transfer() on the local instance

on_connect(conn)[source]¶

on_disconnect(conn)[source]¶

class pylablib.core.utils.rpyc_utils.DeviceService(verbose=False)[source]¶

Bases: pylablib.core.utils.rpyc_utils.SocketTunnelService

Device RPyC service.

Expands on SocketTunnelService by adding get_device() method, which opens local devices, tracks them, and closes them automatically on disconnect.

on_connect(conn)[source]¶

on_disconnect(conn)[source]¶

get_device_class(cls)[source]¶

Get remote device class.

cls is the full class name, including the module within pylablib.devices (e.g., Attocube.ANC300).

get_device(cls, *args, **kwargs)[source]¶

Connect to a device.

cls is the full class name, including the module within pylablib.devices (e.g., Attocube.ANC300). Stores reference to the connected device and closes it automatically on disconnect.

obtain(proxy)¶: Execute obtain() on the local instance

transfer(obj)¶: Execute transfer() on the local instance

tunnel_recv(unpacker=None)¶

Receive data sent through the socket tunnel.

If unpacker is not None, it defines a function to convert the received bytes string into an object.

tunnel_send(obj, packer=None)¶

Send data through the socket tunnel.

If packer is not None, it defines a function to convert obj to a bytes string.

pylablib.core.utils.rpyc_utils.run_device_service(port=18812, verbose=False)[source]¶: Start DeviceService at the given port

pylablib.core.utils.rpyc_utils.connect_device_service(addr, port=18812, timeout=3, attempts=2, error_on_fail=True, config=None)[source]¶

Connect to the DeviceService running at the given address and port

timeout and attempts define respectively timeout of a single connection attempt, and the number of attempts (RPyC default is 3 seconds timeout and 6 attempts). If error_on_fail==True, raise error if the connection failed; otherwise, return None

pylablib.core.utils.strdump module¶

Utils for converting variables into standard python objects (lists, dictionaries, strings, etc.) and back (e.g., for a more predictable LAN transfer). Provides an extension for pickle for more customized classes (numpy arrays, Dictionary).

class pylablib.core.utils.strdump.StrDumper[source]¶

Bases: object

Class for dumping and loading an object.

Stores procedures for dumping and loading, i.e., conversion from complex classes (such as Dictionary) to simple built-in classes (such as dict or str).

add_class(cls, dumpf=None, loadf=None, name=None, allow_subclass=True, recursive=False)[source]¶

Add a rule for dumping/loading an object of class cls.

Parameters:

cls –
dumpf (callable) – Function for dumping an object of the class; None means identity function.
loadf (callable) – Function for loading an object of the class; None means identity function.
name (str) – Name of class, which is stored in the packed data (cls.__name__ by default).
allow_subclass (bool) – If True, this rule is also used for subclasses of this class.
recursive (bool) – If True, the functions are given a second argument, which is a dumping/loading function for their sub-elements.

dump(obj)[source]¶: Convert an object into a dumped value

load(obj)[source]¶: Convert a dumped value into an object

loads(s)[source]¶: Convert a pickled string of a damped object into an object

dumps(obj)[source]¶: Dump an object into a pickled string

pylablib.core.utils.strdump.dumper = <pylablib.core.utils.strdump.StrDumper object>¶

Default dumper for converting into standard Python classes and pickling.

Converts numpy.ndarray and Dictionary objects (these conversion routines are defined when corresponding modules are imported). The converted values include non-printable characters (conversion uses numpy.load() and numpy.ndarray.dump()), so they can’t be saved into text files. However, they’re suited for pickling.

pylablib.core.utils.strdump.dump(obj)[source]¶: Convert obj into standard Python classes using the default dumper

pylablib.core.utils.strdump.load(s)[source]¶: Convert standard Python class representation s into an object using the default dumper

pylablib.core.utils.strdump.dumps(obj)[source]¶: Convert obj into a pickled string using the default dumper

pylablib.core.utils.strdump.loads(s)[source]¶: Convert a pickled string into an object using the default dumper

pylablib.core.utils.string module¶

String search, manipulation and conversion routines.

pylablib.core.utils.string.string_equal(name1, name2, case_sensitive=True, as_prefix=False)[source]¶

Determine if name1 and name2 are equal with taking special rules (case_sensitive and as_prefix) into account.

If as_prefix==True, strings match even if name1 is just a prefix of name2 (not the other wait around).

pylablib.core.utils.string.find_list_string(name, str_list, case_sensitive=True, as_prefix=False, first_matched=False)[source]¶

Find name in the string list.

Comparison parameters are defined in string_equal(). If first_matched==True, stop at the first match; otherwise if multiple occurrences happen, raise ValueError.

Returns:	tuple `(index, value)`.

pylablib.core.utils.string.find_dict_string(name, str_dict, case_sensitive=True, as_prefix=False)[source]¶

Find name in the string dictionary.

Comparison parameters are defined in string_equal(). If multiple occurrences happen, raise ValueError.

Returns:	tuple `(key, value)`.

pylablib.core.utils.string.find_first_entry(line, elements, start=0, not_found_value=-1)[source]¶

Find the index of the earliest position inside the line of any of the strings in elements, starting from start.

If none are found, return not_found_value.

pylablib.core.utils.string.find_all_first_locations(line, elements, start=0, not_found_value=-1, known_locations=None)[source]¶

Find the indices of the earliest position inside the line of all of the strings in elements, starting from start.

Return dict {element: pos}, where pos is either position in the string, or not_found_value if no entries are present. known_locations can specify a dictionary of already known locations of some of the elements. In this case, only missing elements or elements located before start will be re-evaluated.

pylablib.core.utils.string.translate_string_filter(filt, syntax, match_case=True, default=False)[source]¶

Turns filt into a matching function.

The matching function takes single str argument, returns bool value.

filt can be

None: function always returns default,
bool: function always returns this value,
str: pattern, determined by syntax,
anything else: returned as is (assumed to already be a callable).

syntax can be 're' (re), 'glob' (glob) or 'pred' (simply matching predicate). match_case determines whether the filter cares about the string case when matching.

class pylablib.core.utils.string.StringFilter(include=None, exclude=None, syntax='re', match_case=False)[source]¶

Bases: object

String filter function.

Matches string if it matches include (matches all strings by default) and doesn’t match exclude (matches nothing by default).

Parameters:

include – Inclusion filter (translated by translate_string_filter() with syntax specified by syntax); include all by default.
exclude – Exclusion filter (translated by translate_string_filter() with syntax specified by syntax); exclude none by default.
syntax – Default syntax for pattern filters. Can be 're' (re), 'glob' (glob) or 'pred' (simply matching predicate).
match_case (bool) – Determines whether filter ignores case when matching.

pylablib.core.utils.string.get_string_filter(include=None, exclude=None, syntax='re', match_case=False)[source]¶

Generate StringFilter with the given parameters.

If the first argument is already StringFilter, return as is. If it’s a tuple, expand as argument list.

pylablib.core.utils.string.sfglob(include=None, exclude=None)[source]¶: Return string filter based on glob syntax

pylablib.core.utils.string.sfregex(include=None, exclude=None, match_case=False)[source]¶: Return string filter based on re syntax

pylablib.core.utils.string.filter_string_list(l, filt)[source]¶: Filter string list based on the filter

pylablib.core.utils.string.escape_string(value, location='element', escape_convertible=True, quote_type='"')[source]¶

Escape string.

Escaping can be partially skipped depending on location:

"parameter": escape only if it contains hard delimiters ("\n\t\v\r") anywhere

or _border_escaped (", ' or space) on the sides (suited for parameters taking the full string);
"entry": same as above, plus containing soft delimiters (, or space) anywhere (suited for entries of a table);
"element": always escaped

If escape_convertible==True, escape strings which can be misinterpreted as other values, such as "1" or "[]";

otherwise, escape only strings which contain special characters.

If quote_type is not None, automatically put the string into the specified quotation marks;

if quote_type is None, all quotation marks are escaped; if it’s not None, only quote_type marks are escaped.

class pylablib.core.utils.string.TConversionClass(label, cls, rep, conv)¶

Bases: tuple

cls¶

conv¶

count()¶: Return number of occurrences of value.

index()¶

Return first index of value.

Raises ValueError if the value is not present.

label¶

rep¶

pylablib.core.utils.string.add_conversion_class(label, cls, rep, conv)[source]¶

Add a string conversion class.

Some values (e.g., numpy arrays or named tuples) lose some of their associated information when converted into strings. With this function is possible to define custom conversion rules for such classes.

Parameters:

label (str) – class label (e.g., "array")
cls – class which is used to determine if the value should use this conversion functions (e.g., np.ndarray)
rep – function which takes a single argument (object of class cls) and returns its representations; can return a string or an object which is easier to convert to a string (e.g., a list or a tuple)
conv – function which takes one or several arguments (converted values of the class representation) and returns the corresponding object; if rep returns a tuple, treat it as a list of several arguments, which are passed to conv separately; otherwise, conv gets a single argument which is the result of rep

When converting to string, if an object of class cls is encountered, it is converted in a string label(str_rep) (e.g., "array([0, 1, 2])"), where str_rep is the result of calling rep (if this result is a tuple, avoid double parentheses, e.g., if the result is a tuple (1, 2), the string becomes "label(1, 2)" instead of "label((1, 2))"). When converting from string, the values inside the parentheses are passed as arguments to conv function to get the resulting value.

pylablib.core.utils.string.add_namedtuple_class(cls)[source]¶

Add conversion class for a given named tuple class.

For details, see add_conversion_class().

pylablib.core.utils.string.to_string(value, location='element', value_formats=None, parenthesis_rules='text', use_classes=False)[source]¶

Convert value to string with an option of modifying format string.

Parameters:

value –
location (str) – Used for converting strings (see escape_string()).
value_formats (dict) – dictionary {value_type: fmt}, where value type can be int, float or complex and fmt is a format string used to represent value of this type (e.g., "5.3f"); default formats are {float:".12E", complex:".12E", int:"d"}.
parenthesis_rules (str) – determine how to deal with single-element tuples and complex numbers can be "text" (single-element tuples are represented with simple parentheses, e.g., "(1)"; complex number are represented without parentheses, e.g., "1+2j") or "python" (single-element tuples are represented with a comma in the end, e.g., "(1,)"; complex number are represented with parentheses, e.g., "(1+2j)")
use_classes (bool) – if True, use additional representation classes for special objects (e.g., numpy arrays will be represented as "array([1, 2, 3])" instead of just "[1, 2, 3]"). This improves conversion fidelity, but makes result harder to parse (e.g., by external string parsers). See add_conversion_class() for more explanation.

pylablib.core.utils.string.is_convertible(value)[source]¶: Check if the value can be converted to a string using standard to_string() function.

pylablib.core.utils.string.extract_escaped_string(line, start=0)[source]¶

Extract escaped string in quotation marks from the line, starting from start.

line[start] should be a quotation mark (' or ") or r or b followed by a quotation mark (for raw or binary strings).

Returns:	tuple `(end position, un-escaped string)`.

pylablib.core.utils.string.unescape_string(value)[source]¶

Un-escape string.

Only attempt if the string starts a quotation mark " or '. Otherwise (including strings like 'r""' or 'b""'), return the string as is. Raise an error if the string starts with a quotation mark, but does not correspond to a proper escaped string (e.g., '"abc or '"abc"def).

pylablib.core.utils.string.to_range(range_tuple)[source]¶

pylablib.core.utils.string.from_string(value, case_sensitive=True, parenthesis_rules='text', use_classes=True)[source]¶

Parse a string.

Recognizes integers, floats, complex numbers (with i or j for complex part), strings (in quotation marks), dicts, sets, list and tuples, booleans and None. If item is unrecognizable, assumed to be a string.

Parameters:

case_sensitive (bool) – applied when compared to None, True or False.
parenthesis_rules (str) –
determines how to deal with empty entries (e.g., [1,,3]) and complex number representation ("1+2j" vs. "(1+2j)"):
- 'text': any empty entries are translated into empty_string (i.e., [,] -> [empty_string, empty_string]), except for completely empty structures ([] or ()); complex numbers are represented without parentheses, so that "(1+2j)" will be interpreted as a single-element tuple (1+2j,).
- 'python': empty entries in the middle are not allowed; empty entries at the end are ignored (i.e., [2,] -> [2]) (single-element tuple can still be expressed in two ways: (e,) or (e)); complex numbers are by default represented with parentheses, so that "(1+2j)" will be interpreted as a complex number, and only (1+2j,), ((1+2j)) or ((1+2j),) as a single-element tuple.
use_classes (bool) – if True, use additional representation classes for special objects (e.g., "array([1, 2, 3])" will be converted into a numpy array instead of raising an error). See add_conversion_class() for more explanation.

pylablib.core.utils.string.from_string_partial(value, delimiters=re.compile('\\s*, \\s*|\\s+'), case_sensitive=True, parenthesis_rules='text', use_classes=True, return_string=False)[source]¶

Convert the first part of the supplied string (bounded by delimiters) into a value.

delimiters is a string or a regexp (default is "\s*,\s*|\s+", i.e., comma or spaces). If return_string==False, convert the value string and return tuple (end_position, converted_value); otherwise, return tuple (end_position, value_string).

The rest of the parameters is the same as in from_string().

pylablib.core.utils.string.from_row_string(value, delimiters=re.compile('\\s*, \\s*|\\s+'), case_sensitive=True, parenthesis_rules='text', use_classes=True, return_string=False)[source]¶

Convert the row string into a list of values, separated by delimiters.

If return_string==False, return list of converted objects; otherwise, return list of unconverted strings.

The rest of the parameters is the same as in from_string_partial().

pylablib.core.utils.strpack module¶

Utilities for packing values into bitstrings. Small extension of the struct module.

pylablib.core.utils.strpack.int2bytes(val, l, bo='>')[source]¶

Convert integer into a list of bytes of length l.

bo determines byte order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.bytes2int(val, bo='>')[source]¶

Convert a list of bytes into an integer.

bo determines byte order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.int2bits(val, l, bo='>')[source]¶

Convert integer into a list of bits of length l.

bo determines byte (and bit) order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.bits2int(val, bo='>')[source]¶

Convert a list of bits into an integer.

bo determines byte (and bit) order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.pack_uint(val, l, bo='>')[source]¶

Convert an unsigned integer into a bytestring of length l.

Return bytes object. bo determines byte order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.pack_int(val, l, bo='>')[source]¶

Convert a signed integer into a bytestring of length l.

Return bytes object. bo determines byte order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.unpack_uint(msg, bo='>')[source]¶

Convert a bytestring into an unsigned integer.

bo determines byte order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.unpack_int(msg, bo='>')[source]¶

Convert a bytestring into an signed integer.

bo determines byte order: '>' is big-endian (MSB first), '<' is little-endian (LSB first).

pylablib.core.utils.strpack.unpack_numpy_u12bit(buffer, byteorder='<', count=-1)[source]¶

pylablib.core.utils.units module¶

Routines for conversion of physical units.

pylablib.core.utils.units.split_units(value)[source]¶

Split string value with a dimension.

Return tuple (val, unit), where val is the float part of the value, and unit is the string representing units.

pylablib.core.utils.units.convert_length_units(value, value_unit='m', result_unit='m', case_sensitive=True)[source]¶

Convert value from value_unit to result_unit.

The possible length units are 'm', 'mm', 'um', 'nm', 'pm', 'fm'. If case_sensitive==True, matching units is case sensitive.

pylablib.core.utils.units.convert_time_units(value, value_unit='s', result_unit='s', case_sensitive=True)[source]¶

Convert value from value_unit to result_unit.

The possible time units are 's', 'ms', 'us', 'ns', 'ps', 'fs', 'as'. If case_sensitive==True, matching units is case sensitive.

pylablib.core.utils.units.convert_frequency_units(value, value_unit='Hz', result_unit='Hz', case_sensitive=True)[source]¶

Convert value from value_unit to result_unit.

The possible frequency units are 'Hz', 'kHz', 'MHz', 'GHz'. If case_sensitive==True, matching units is case sensitive.

pylablib.core.utils.units.convert_power_units(value, value_unit='dBm', result_unit='dBm', case_sensitive=True, impedance=50.0)[source]¶

Convert value from value_unit to result_unit.

For conversion between voltage and power, assume RMS voltage and the given impedance. The possible power units are 'dBm', 'dBmV', 'dBuV', 'W', 'mW', 'uW', 'nW', 'mV', 'nV'. If case_sensitive==True, matching units is case sensitive.

pylablib.core.utils package¶

Submodules¶

pylablib.core.utils.array_utils module¶

pylablib.core.utils.ctypes_wrap module¶

pylablib.core.utils.dictionary module¶

pylablib.core.utils.files module¶

pylablib.core.utils.funcargparse module¶

pylablib.core.utils.functions module¶

pylablib.core.utils.general module¶

pylablib.core.utils.indexing module¶

pylablib.core.utils.ipc module¶

pylablib.core.utils.library_parameters module¶

pylablib.core.utils.module module¶

pylablib.core.utils.nbtools module¶

pylablib.core.utils.net module¶

pylablib.core.utils.numerical module¶

pylablib.core.utils.observer_pool module¶

pylablib.core.utils.py3 module¶

pylablib.core.utils.rpyc_utils module¶

pylablib.core.utils.strdump module¶

pylablib.core.utils.string module¶

pylablib.core.utils.strpack module¶

pylablib.core.utils.units module¶

Module contents¶