XSVWriter#
- class streaming.XSVWriter(dirname, columns, separator, compression=None, hashes=None, size_limit=67108864, newline='\n')[source]#
Writes a streaming XSV dataset.
- Parameters
dirname (str) – Local dataset directory.
separator (str) – String used to separate columns.
compression (str, optional) – Optional compression or compression:level. Defaults to
None
.hashes (List[str], optional) – Optional list of hash algorithms to apply to shard files. Defaults to
None
.size_limit (int, optional) – Optional shard size limit, after which point to start a new shard. If None, puts everything in one shard. Defaults to
None
.newline (str) – Newline character inserted between samples. Defaults to
\\n
.
- encode_sample(sample)[source]#
Encode a sample dict to bytes.
- Parameters
sample (Dict[str, Any]) – Sample dict.
- Returns
bytes – Sample encoded as bytes.