DataConverterAdaptor

class servicex.data_conversions.DataConverterAdaptor(default_file_type: str)

Bases: object

Methods to convert from one type of data to the other.

__init__(default_file_type: str)

Create a data converter adaptor. By default it will do the conversation as requested.

Args:

default_file_type (str): The default file type (parquet or root)

Methods

async _convert_parquet_to_awkward(file: Path)

Convert the contents of a parquet file to an awkward dictionary.

Arguments:

file A Path to the file containing the pandas data

Returns:

DataFrame A pandas DataFrame

Note:

• Work is done on a second thread.

• Pandas is only imported if this is called.

async _convert_parquet_to_pandas(file: Path)

Convert the contents of a parquet file to pandas.

Arguments:

file A Path to the file containing the pandas data

Returns:

DataFrame A pandas DataFrame

Note:

• Work is done on a second thread.

• Pandas is only imported if this is called.

async _convert_root_to_awkward(file: Path)

Convert the contents of a ROOT file to an awkward dictionary.

Arguments:

file A Path to the file containing the pandas data

Returns:

DataFrame A pandas DataFrame

Note:

• Work is done on a second thread.

• Awkward is only imported if this is called.

• A LazyArray is returned, so it isn’t completely loaded into memory. That also means this will leak file handles - as that has to be left open.

async _convert_root_to_pandas(file: Path)

Convert the contents of a ROOT file to pandas.

Arguments:

file A Path to the file containing the pandas data

Returns:

DataFrame A pandas DataFrame

Note:

• Work is done on a second thread.

• Pandas is only imported if this is called.

combine_awkward(awks: Iterable[Array]) → Array

Combine many awkward arrays into a single one, in order.

Args:

awks (Iterable[ChunkedArray]): The input list of awkward arrays

combine_pandas(dfs: Iterable[DataFrame]) → DataFrame

Combine many pandas DataFrame into a single one, in order.

Args:

dfs (Iterable[pd.DataFrame]): The list of DataFrames

async convert_to_awkward(file: Path, file_type: Optional[str] = None)

Convert to an awkward data array from data stored in a file of a particular file_type

Args:

file (Path): Path to the file file_type (str): What the file contains (root, parquet, etc)

async convert_to_pandas(file: Path, file_type: Optional[str] = None)

Convert to a pandas DataFrame from data stored in a file of a particular file_type

Args:

file (Path): Path to the file file_type (str): What the file contains (root, parquet, etc)