Pipelines - PGFS

Pipelines has a "Postgres File System" (PGFS) extension that allows you to access external storage. In particular, it allows you to store and retrieve files from either a public or private S3-compatible bucket or a local file system. The aidb extension addresses external storage mapped through pgfs as "storage locations" and abstracts away the details of the underlying storage.

PGFS is a key component of Pipelines, allowing you to work with data which is not stored in the database, either because it is too large or because it is not structured in a way that is suitable for a relational database.

PGFS is implemented as a foreign data wrapper (FDW) and is installed as an extension. The extension provides a set of functions to interact with the file system from within the database.

Installing the PGFS extension

See Installing for instructions on how to install the PGFS extension as part of the Pipelines installation.

Using PGFS

There are two approaches to using PGFS.

• (Recommended) Use the PGFS functions to create and delete storage locations

• Use SQL commands such as CREATE SERVER command to create a server object that creates the storage location.

Using the PGFS functions

See the PGFS functions reference for details on the functions available in the PGFS extension.

Using PGFS with SQL

See the PGFS SQL reference for details on how to use PGFS with SQL commands.