LocalDataset

class streaming.LocalDataset(local, split=None)

A streaming dataset whose shards reside locally as a pytorch Dataset.

Parameters

• local (str) – Local dataset directory where shards are cached by split.

• split (str, optional) – Which dataset split to use, if any. Defaults to None.