Advanced Python API usage¶

Typing of dataframes¶

Applies when reading a dataframe.

By default, the data frame is created without explicit typing. This means that we let Pandas “guess” the proper Pandas type for each column. The main advantage of this approach is that even if your dataset only contains “string” column (which is the default on a newly imported dataset) , if the column actually contains numbers, a proper numerical type will be used.

If you pass infer_with_pandas=False as option to get_dataframe(), the exact dataset types will be passed to Pandas. Note that if your dataset contains invalid values, the whole get_dataframe call will fail.

Chunked reading and writing with Pandas¶

When using Dataset.get_dataframe(), the whole dataset (or selected partitions) are read into a single Pandas dataframe, which must fit in RAM on the DSS server.

This is sometimes inconvenient and DSS provides a way to do this by chunks:

mydataset = Dataset("myname")

for df in mydataset.iter_dataframes(chunksize=10000):
        # df is a dataframe of at least 10K rows.

By doing this, you only need to load a few thousands of rows at a time.

Writing in a dataset can also be made by chunks of dataframes. For that, you need to obtain a writer:

inp = Dataset("input")
out = Dataset("output")

with out.get_writer() as writer:

        for df in inp.iter_dataframes():
                # Process the df dataframe ...

                # Write the processed dataframe
                writer.write_dataframe(df)

Encoding¶

When dealing with both dataframes and row-by-row iteration, you must pay attention to str/unicode and encoding issues

• DSS provides dataframes where the string content is utf-8 encoded str
• When writing dataframes, DSS expects utf-8 encoded str
• Per-line iterators provide string content as unicode objects
• Per-line writers expect unicode objects.

For example, if you read from a dataframe but write row-by-row, you must decode your str into Unicode object

Sampling¶

All calls to iterate the dataset (get_dataframe, iter_dataframes, iter_rows and iter_tuples) take several arguments to set sampling.

Sampling lets you only retrieve a selection of the rows of the input dataset. It’s often useful when using Pandas if your dataset does not fit in RAM.

For more information about sampling methods, please see Sampling.

The sampling argument takes the following values.

# Get a Dataframe over the first 3K rows
dataset.get_dataframe(sampling='head', limit=3000)

# Iterate over a random 10% sample
dataset.iter_tuples(sampling='random', ratio=0.1)

# Iterate over 27% of the values of column 'user_id'
dataset.iter_tuples(sampling='random-column', sampling_column='user_id', ratio=0.27)

# Get a chunked stream of dataframes over 100K randomly selected rows
dataset.iter_dataframes(sampling='random', limit=100000)

Getting a dataset as raw bytes¶

In addition to retrieving a dataset as Pandas Dataframes or iterator, you can also ask DSS for a streamed export, as formatted data.

Data can be exported by DSS in various formats: CSV, Excel, Avro, ...

# Read a dataset as Excel, and dump to a file, chunk by chunk
#
# Very important: you MUST use a with() statement to ensure that the stream
# returned by raw_formatted is closed

with open(target_path, "w") as ofl:
        with dataset.raw_formatted_data(format="excel") as ifl:
                while True:
                        chunk = ifl.read(32000)
                        if len(chunk) == 0:
                                break
                        ofl.write(chunk)

See dataiku.Dataset for complete API documentation.