Flo.w

# Dataset Types

Flo.w Engine provides a number of dataset types to support different use-cases and data sources. Irrespective of the type, datasets provide a unified API and can generally be used transparently by other components of the Flo.w ecosystem.

The type of a dataset is specified by its type property. This determines how the dataset's source property will be interpreted.

# Table Datasets

Table datasets are the simplest type of dataset and expose data from a single database table. To configure a table dataset set:

  • type: table.
  • source: The name of the associated database table.

Querying a table dataset is equivalent to querying the associated database table.

# SQL Datasets

SQL datasets are equivalent to database views and are defined by a SQL select expression. To configure a SQL dataset set:

  • type: sql.
  • source: A SQL select expression (without trailing semicolon).

When querying a SQL dataset, the specified SQL source is combined as a sub-query with other Flo.w dataset query operators (such as limit, where, order and attributes).

You may prefer to use SQL datasets:

  • to move query complexity to SQL and simplify using the dataset from front-end applications.
  • to specify data joins between tables.
  • to use database-specific query functionality that is not supported by Flo.w query definitions.
  • if you are more comfortable with SQL than Flo.w query definitions.

# Parameterized Datasets

Parameterized datasets are a sub-type of SQL datasets that include dynamic parameters in the SQL expression. Dynamic parameters are specified in the SQL expression using placeholders and are supplied when querying the dataset.

An example SQL expression with a name dynamic parameter no2_filter:

SELECT * from airquality WHERE no2 > {{no2_filter}}::double precision order by no2 desc

See Querying Parameterized Datasets for details of how to query parameterized datasets and how the parameters are evaluated.

# External Datasets

Flo.w Engine also provides datasets that are backed by external data. This provides a mechanism for integrating 3rd-party services in a unified manner that can be used by Flo.w applications along with Flo.w authentication.

WARNING

Flo.w query definitions cannot be used with external datasets. Query capability depends on the external data source and can be configured by passing query parameters only.

# REST Datasets

REST datasets are backed by a request to an external REST endpoint. Querying a REST dataset will initiate a request to the associated 3rd-party service. Query parameters are added to the query string of the configured REST URL.

To configure a REST dataset set:

  • type: rest.
  • source: The 3rd-party REST URL.
  • options.rest.headers: Additional HTTP headers to send with the request:
    • e.g. {"x-access-token": "abcd-1234"}

# AWS Lambda Datasets

AWS Lambda datasets are backed by a request to an AWS Lambda function. Querying an AWS Lambda dataset will initiate a request to the associated function. The query definition and additional query parameters are added to the request and can be extracted in the lambda function. Note that it is the responsibility of the Lambda Function to interpret and apply any supported components of the query definition (for example, the limit option).

AWS Lambda datasets can be used to access 3rd-party services that cannot be accessed with a simple REST request. For instance, the service may require that a short-lived access token is obtained and then sent with a subsequent request.

To configure an AWS Lambda dataset set:

  • type: awslambda.
  • source: The ARN of the Lambda function.
  • options.awsLambda: AWS credential and options:
    • e.g. {"region": "eu-west-1", "accessKeyId": "xxxxx", "secretAccessKey", "xxxx"}

Access credentials can also be configured at Flo.w Engine server level to avoid exposing the credentials in the dataset configuration.

The Lambda function handler should return an object indicating the status of the request and the resulting data as an array of data items. An example handler is shown below:

// Example AWS Lambda handle function for a Lambda dataset
//
exports.handler = async (event) => {

  // Query parameters can be retrieved from event object
  console.log('Request params', event.params);

  try {
    // Get data - do something interesting here
    const data = [{id: 1, val: 'val1'}, {id: 2, val: 'val2'];

    // Indicate success and return data
    return {
      status: 200,
      data
    };
  } catch (e) {
    // Indicate error
    return {
      status: e.status || 500,
      message: e.message,
      details: 'Additional error details'
    };
  }
};

← Uploading Data →