Data sources
Delta sources experimental data through data sources. An instantiated data source object is used in generators to load chunks of experimental data and stage it for transport. Here is an example of how data sources are used:
1 2 3 4 5 | sources.helpers import get_loader
dataloader = get_loader(cfg)
for nstep, chunk in dataloader.batchgenerator()
do_things_with(chunk)
|
Dataloader objects are generated by the sources.helpers.get_loader() object factory,
passing only the Delta configuration to the factory. Internally, sources.helpers.get_loader()
queries the field diagnostic.name to select the proper dataloader. Implementing a new dataloader requires
to register it in get_loader.
To be useful in a streaming setting, any batchgenerator() returns the instances of a data_model.