# Reference Deployment Guide

# Introduction

This document describes how to deploy Flo.w Engine, the core of the Flo.w ecosystem. A reference deployment is also provided, which serves as a starting point and learning guide for moving to a full production-quality cloud-based deployment. A Reference Deployment Pack is available separately and provides the files described in the Reference Deployment section.

Before starting, it is worth providing some context. What is Flo.w Engine and where does it sit in the Flo.w ecosystem?

# Context

Flo.w Engine is the core of the Flo.w ecosystem and provides base services to Flo.w apps:

Authentication/authorisation
Geospatial data management and storage
Geospatial data file upload supporting all common (and uncommon) formats e.g. ESRI Shapefile, GeoJSON, CSV
Dataset querying
Map vector tile generation

Optional components also support realtime ingestion of data from a wide variety of sources using both data push and pull mechanisms.

Deployment Architecture

The diagram above illustrates the Flo.w ecosystem. Flo.w Engine sits at the centre and is highlighted in blue.

The engine connects to a PostgreSQL database server (v10 or above) to store its management and metadata database. Additional per-application databases are also used to store application-specific geospatial data. This ensures a clean separation between applications, and between application data and Flo.w Engine’s own management data. Flo.w engine also has the capability of connecting to other database servers and external REST APIs through a data abstraction layer.

The engine also connects to a Redis cache, which it uses to improve map performance.

Flo.w applications are typically front-end only or front-end/back-end web apps that interface with Flo.w Engine through its REST API (exposed on port 3000 by default). Flo.w apps can also request vector map tiles for displaying interactive maps through the same port.

To the right of Flo.w Engine in the diagram is the realtime data ingest system and an optional ETL/Analyst environment. The ETL/Analyst environment provides a secure workspace for bulk data preparation and loading. Tools required by analysts are installed within this environment to allow analysts to work with sensitive data and prepare datasets for upload to Flo.w Engine or direct loading into an application’s database.

# Reference Deployment

The reference deployment is a minimal deployment of Flo.w Engine and supporting services that can be run on a developer’s laptop. Developers wishing to deploy Flo.w Engine should work through the minimal reference deployment to familiarize themselves with the system before attempting to deploy Flo.w engine in a cloud environment and/or scale up the system.

The reference deployment is based on a simple docker-compose file that brings up Flo.w Engine, a Postgres 12 database with PostGIS extensions, and a Redis map tile cache. Despite its simplicity, the reference deployment is a fully-working Flo.w Engine system and can be used for local development and testing of Flo.w applications.

# Prerequisites

To follow the reference deployment you will need:

Docker Community Edition v17.12.0 or greater
docker-compose v1.25 or greater
Node.js & npm v10 or greater (optional)

To install Docker and docker-compose on Mac, Linux or Windows, please refer to Get Docker (opens new window) on the Docker website.

Node.js and npm (Node Package Manager) are not strictly required to complete the reference deployment but are needed to run the Flo.w command line tool (CLI) to fully test the deployment and will be required to administer Flo.w Engine.

If you don’t have Node.js installed already, we recommend using nvm (opens new window) on Mac and Linux, and nvm-windows (opens new window) for Windows.

# The Reference Deployment Pack

The reference deployment pack contains all the resources you will need to complete the reference deployment:

docker-compose.yml - the Docker Compose configuration file
data/
- fonts - map fonts in SDF format
- sprites - map sprites for point-of-interest markers (for base maps)
- mbtiles - OpenStreetMap world extract in mbtiles format (for base maps)
- uploads - temporary directory to hold geospatial data files during HTTP upload to Flo.w Engine

For reference, the docker-compose.yml is shown below. Please use the version you receive in your Reference Deployment Pack as there may be minor changes.

version: "3.4"

services:
  flow-engine:
    image: 809728736372.dkr.ecr.eu-west-1.amazonaws.com/flow:latest
    ports:
      - 3000:3000
    volumes:
      - ./data:/data
    networks:
      - flow
    environment:
      - FLOW_MASTER_KEY=994cca45-b909-44da-ba82-b23cd51c42d7
      - NODE_ENV=development
      - FLOW_LOG_LEVEL=debug
      - FLOW_SERVER_BASE_URL=http://localhost:3000
      - FLOW_SERVER_SHARED_RESOURCE_BASE_URL=https://flow.emu-analytics.net
      # DB config
      - FLOW_DB_HOST=postgis
      - FLOW_DB_PORT=5432
      - FLOW_DB_DATABASE=flow-engine
      - FLOW_DB_USER=flow-engine
      - FLOW_DB_PASSWORD=secretdbpassword
      # SMTP config
      - FLOW_SMTP_HOST=email-smtp.eu-west-1.amazonaws.com
      - FLOW_SMTP_PORT=587
      - FLOW_SMTP_SECURE=false
      - FLOW_SMTP_USER=**************
      - FLOW_SMTP_PASSWORD=*************
      # Tile cache config
      - FLOW_TILE_CACHE_STORE=redis
      - FLOW_TILE_CACHE_HOST=redis
      - FLOW_TILE_CACHE_PORT=6379
      # Data paths
      - FLOW_FONTS_DIR=/data/fonts
      - FLOW_SPRITES_DIR=/data/sprites
      - FLOW_FILE_UPLOAD_PATH=/data/uploads

  postgis:
    image: postgis/postgis:11-2.5
    ports:
      - 5433:5432
    networks:
      - flow
    environment:
      - POSTGRES_DB=flow-engine
      - POSTGRES_USER=flow-engine
      - POSTGRES_PASSWORD=secretdbpassword

  redis:
    image: redis
    networks:
      - flow

networks:
  flow:

# Starting the System

The Reference Deployment Pack contains everything in place to start a running system. Bring up the Docker Compose stack by issuing the following command in the root directory of the pack:

docker-compose up

This will start the three services defined in the stack and leave your terminal attached to receive logging messages. If successful, you should see the following as the last output from the start-up sequence:

flow-engine_1  | 2020-06-24T15:15:28.973Z - [flow-engine] info: Flow DB connected {"type":"postgres","host":"postgis","port":"5432","username":"flow-engine","password":"*******","database":"flow-engine","synchronize":false,"migrationsRun":true,"logging":false}
flow-engine_1  | 2020-06-24T15:15:28.974Z - [flow-engine] info: Registered database driver 'postgis-mapnik'
flow-engine_1  | 2020-06-24T15:15:28.974Z - [flow-engine] info: Registered database driver 'postgis'
flow-engine_1  | 2020-06-24T15:15:29.112Z - [flow-engine] info: Loaded fonts: /data/fonts
flow-engine_1  | 2020-06-24T15:15:29.112Z - [flow-engine] debug: Nest application successfully started {"context":"NestApplication"}
flow-engine_1  | 2020-06-24T15:15:29.115Z - [flow-engine] info: App listening on port 3000: Version: v1.0.11 (build time: 2020-06-24T14:16:36.000Z)

Note that the first time you run the stack, the postgres service takes some time to initialize and create the default database and user. You will see errors from the flow-engine service while this is happening but the service will eventually connect to the database and continue initialization. You should not see this error on subsequent runs of the stack.

flow-engine_1  | 2020-06-24T15:15:25.204Z - [flow-engine] error: Unable to connect to the database. Retrying (2)... {"trace":"Error: connect ECONNREFUSED 172.27.0.4:5432

# Testing the System

In a separate terminal, issue the following command to hit the config endpoint of Flo.w Engine’s REST API:

curl -H x-api-key:994cca45-b909-44da-ba82-b23cd51c42d7 \
  http://localhost:3000/api/v1/config

If the system is running correctly you should see the following response:

{"version":{"tag":"v1.0.11","hash":"1557ec9","timestamp":1593008196}}

Optionally, you can also test using the Flo.w command line tool if you have installed the Node.js prerequisites described above. Run the following command once to install the CLI globally from npm:

npm install -g flow-engine-cli

Once installed the Flo.w CLI is available using the flow command. (Hint: run flow --help). To test connection to the running Flo.w Engine issue the following command:

flow version -k 994cca45-b909-44da-ba82-b23cd51c42d7

You should see the following output:

╔═══════════════╤═══════════════════════════╗
║ Property      │ Value                     ║
╟───────────────┼───────────────────────────╢
║ CLI version   │ 1.0.5                     ║
╟───────────────┼───────────────────────────╢
║ Flo.w version │ v1.0.11                   ║
╟───────────────┼───────────────────────────╢
║ Flo.w host    │ http://localhost:3000     ║
╟───────────────┼───────────────────────────╢
║ API key       │ 994cca45-b909-44da-ba82-b ║
║               │ 23cd51c42d7               ║
╚═══════════════╧═══════════════════════════╝

# Controlling the Stack

The command docker-compose up starts the stack and leaves your terminal attached to receive logging output. Press ctrl-c to exit. This will also shutdown the running services. The following commands can then be used to control the stack:

# Bring the stack back up and attach terminal
docker-compose up

# Bring the stack up in the background (daemon mode)
docker-compose up -d
docker-compose logs -f # Tail log messages

# Stop the stack (when running in background daemon mode)
docker-compose stop

# Remove stack to start from scratch (warning: all database data is lost!)
docker-compose down

# What Just Happened?

You have now got a minimal running Flo.w Engine system. The docker-compose.yml started three required services:

flow-engine - the Flo.w Engine service is the core of the Flo.w ecosystem and provides base functionality required by all Flo.w apps: geospatial data management and storage, authentication and authorization, dataset querying and map tile generation.

postgis - The standard docker image is used to start a Postgres 11.10 database server instance with PostGIS 2.5.5 extensions. This is Flo.w Engine’s standard backend database, although other database servers are supported. A database and database user is also created the first time the service is run.

redis - The standard docker image is used to start a Redis cache. Redis is used by Flo.w engine to cache map tiles that it generates to accelerate map drawing and reduce workload.

# Connectivity

The docker-compose.yml file also defines a Docker bridge network called flow to provide connectivity between the three services. Environment variables are defined to pass Postgres and Redis connection information to the flow-engine service.

Port 3000 of the flow-engine service is mapped to port 3000 of the host system to allow connection to its REST API.

Port 5432 of the postgis service is mapped to port 5433 of the host system so that you can connect a Postgres Client tool to inspect the database. Note the non-standard host port 5433. This is to avoid clashing with any Postgres server you may have running already on your development system. You may change the docker-compose.yml file to expose the PostgreSQL on the standard 5432 port if you wish.

The redis service is only required internally by Flo.w Engine and so its port is not mapped to the host system.

# Volume mounting

The Flo.w Engine expects to be provided static data such as fonts, sprites and base map tiles. To reduce image size and to allow multiple Docker containers to share the same static data, Docker volume mounting is used to mount the static data from the host file system into the running Docker container. The Reference Deployment mounts the static data from the data directory of the Reference Deployment Pack and uses environment variables to configure Flo.w Engine to locate the data.

For more information about how the flow-engine service is configured see the following section.

# Flo.w Engine Configuration

By default, Docker environment variables are used to configure Flo.w Engine. These configuration parameters specify behavior of the system, location of required static data and connectivity to required services.

The configuration system can integrate with Docker Secrets rather than using environment variables, if required. This is beyond the scope of this document. Please contact us for more information on how to use Docker Secrets to configure the system.

Environment Variable	Default	Comment
Master Authentication
`FLOW_MASTER_KEY`	none - must be set	This env variable must be set to a valid UUIDv4 to use as the master API key. This is used to create an initial master key during the first run of the service when the initial DB schema is created and populated. After the initial run of the service this env variable may be removed.
General Configuration
`NODE_ENV`	development	Should be set to production for a production system. Logging messages are produced in JSON format in production mode and a simplified text format in development mode.
`FLOW_LOG_LEVEL`	verbose	Set to info in production to limit logging output to info, warning and error levels.
`FLOW_SERVER_BASE_URL`	http://localhost:3000	Set to the full URL of the Flo.w Engine service as seen by REST API clients. For example, if the service is behind a load balancer at flow-engine.domain.internal that performs SSL/TLS termination and maps port 443 to the internal port 3000 then FLOW_SERVER_BASE_URL should be set to https://flow-engine.internal
`FLOW_SERVER_SHARED_RESOURCE_BASE_URL`	https://flow.emu-analytics.net	Set to the URL of an external Flo.w instance to use for shared resources such as OpenStreetMap world extract data, map fonts and map tiles. Setting this environment variable means that shared resources do not have to be installed locally.
Database Configuration
`FLOW_DB_HOST`	localhost	The Postgres DB server host (as seen from the flow-engine Docker container)
`FLOW_DB_PORT`	5432	The Postgres DB server port (as seen from the flow-engine Docker container)
`FLOW_DB_DATABASE`	flow-engine	The Flo.w Engine management database name
`FLOW_DB_USER`	flow-engine	The Flo.w Engine management database user name
`FLOW_DB_PASSWORD`	none - must be set	The Flo.w Engine management database password
SMTP Server Configuration
`FLOW_SMTP_HOST`	email-smtp.eu-west-1.amazonaws.com	The SMTP server host (as seen from the flow-engine Docker container)
`FLOW_SMTP_PORT`	587	The SMTP server port (as seen from the flow-engine Docker container)
`FLOW_SMTP_SECURE`	false	If true the connection will use TLS when connecting to server. If false (the default) then TLS is used if the server supports the STARTTLS extension.
`FLOW_SMTP_USER`	none - must be set*	The SMTP server user name
`FLOW_SMTP_PASSWORD`	none - must be set*	The SMTP server password
Tile Cache Configuration
`FLOW_TILE_CACHE_STORE`	none	Set to redis to use a Redis tile cache
`FLOW_TILE_CACHE_HOST`	none - must be set**	If using a Redis tile cache set to the Redis host (as seen from the flow-engine Docker container)
`FLOW_TILE_CACHE_PORT`	none - must be set**	If using a Redis tile cache set to the Redis port (as seen from the flow-engine Docker container)
Static Data Configuration
`FLOW_FONTS_DIR`	none - must be set	The static fonts directory within the flow-engine Docker container. (Typically mapped to an external shared volume)
`FLOW_SPRITES_DIR`	none - must be set	The static sprites directory within the flow-engine Docker container. (Typically mapped to an external shared volume)
HTTP File Upload Support
`FILE_UPLOAD_PATH`	none - must be set	The location of the temporary directory for HTTP file upload within the flow-engine Docker container. (Typically mapped to an external shared volume)

* An SMTP server is used to send outgoing emails for user account creation, email address confirmation and password reset. The SMTP username and password can be left undefined if outgoing email is not required. Note that all application-level user management will then need to be performed as a manual administration task.

** Configuring a Redis tile cache is optional but map performance will be severely degraded.

# Moving to Production

The Reference Deployment is a minimal working Flo.w Engine system that can be run on a developer’s laptop. While it illustrates all of the functionality of Flo.w Engine, a production deployment will require some additional planning and work to cover:

Deployment in the cloud
Horizontal scaling
High-availability
Development/test/production environments

# Deployment in the Cloud

Flo.w Engine and consumer-facing Flo.w apps are provided as standard Docker images. Any cloud service that can run Docker containers can be used to deploy Flo.w including but not limited to:

AWS EC2 with bare-bones Docker or Docker Swarm
AWS ECS - with provisioned EC2 instances or serverless Fargate deployment
Azure

Flo.w Engine typically requires a Postgres v10+ database server with PostGIS extensions to host the Flo.w Engine management database and application-specific geospatial databases. All major cloud providers support suitable unmanaged or managed Postgres server instances. For example,

AWS RDS managed Postgres
AWS RDS Aurora serverless Postgres

The database server should be configured and running before attempting to start the Flo.w Engine service. You should also create a new database user (typically called flow-engine) and a new database for Flo.w engine to store its management and metadata information (also typically called flow-engine). The database should be owned by the new user you create and the user should have full access permissions to the database (read, write, DDL). Flo.w engine will create an initial schema and populate it with initial data when it runs for the first time if the database is empty.

Note that Flo.w Engine requires write access to its database so it must be connected to a read/write PostgreSQL instance not a read-only replica if you are using a clustered database server. Flo.w applications that require read-only access may be configured to use read-only replicas for their application-specific databases.

Flo.w Engine benefits from a Redis instance for caching map tiles. This is optional but highly advised. All major cloud providers support suitable managed Redis instances and clusters. For example,

AWS Elasticache Redis

Please refer to your cloud provider documentation for information on how to define and configure the required services.

Within your cloud deployment design you must also consider network connectivity, routing and firewalls. If you wish your Flo.w Engine REST API to be visible externally then you will need to provide a load balancer or application gateway that can perform SSL/TLS termination and map the externally connection to port 3000 of the Flo.w Engine REST API.

If you are planning to host Flo.w apps completely within your cloud infrastructure then the Flo.w Engine REST API should not be exposed externally.

# Horizontal Scaling

Flo.w Engine conforms to the ‘12-factor’ principal to allow the service to be horizontally scaled. Multiple instances of the service can be run behind a load balancer and can share static data. Note that in a horizontally-scaled deployment, the temporary file upload directory (defined by FILE_UPLOAD_PATH) must be mounted as a shared external volume. This is because the Flo.w Engine instance that initially accepts the uploaded file may not be the same instance that subsequently processes it.

# High Availablility

As an extension of horizontal scaling, multiple instances of the Flo.w Engine service can be placed in different availability zones of your cloud provider to mitigate infrastructure failure. Note that the shared static data and temporary file upload directory may also be placed on a high-availability volume (e.g. AWS EFS).

# Development/Test/Production Environments

You may wish to deploy completely separate environments for development, test and production. Alternatively you may choose to use a single database server containing separate databases for each environment and rely on your cloud provider’s high-availability features for database servers.

← Flo.w REST API Reference