Export data to a warehouse

Scheduled, multi-resource snapshots delivered as CSV with a manifest — into Pallara-managed storage or your own S3-compatible bucket.

Staff portalWho: Administrators and analystsLast updated 3 September 2026

Flow

From a destination to a loadable snapshot

Staff actionDecisionStop / refusedDocument / file

A destination that has not passed a connection test cannot run an export. The manifest is written after every part file, so a manifest you can read is a run whose files are all there.

A warehouse export writes whole datasets on a schedule, into storage you can point a warehouse or BI tool at. It is the right tool when something needs all of a dataset, repeatedly — a nightly load into BigQuery, a Power BI refresh, a data-team pipeline. For a question asked once, use the API or an assistant instead.

NoteOne file set per resource, plus a manifest — not one enormous joined CSV. A run writes <resource>/part-000001.csv, part-000002.csv and so on, 500 rows per part, then a manifest.json naming every part with its row count and SHA-256 checksum. The manifest is written last, so a manifest you can read is a run whose files are all there.

Turn it on

Under Administration → Modules, enable Data Warehouse Export.
Administration → Data Warehouse appears — or press ⌘K and type data warehouse.

Creating and editing destinations requires a tenant administrator. Creating and running export sets requires reports-write, which is manager and above. Warehouse runs are a cross-module extract — one file can hold finance, wellbeing and communications columns at once — so read access is deliberately higher than for an ordinary report.

1. Add a destination

A destination is where files land. There are two kinds.

Type	What it is	You supply
Pallara-managed	Storage Pallara runs. Fastest to start; nothing to configure.	A name, and optionally a prefix
S3-compatible	Your own bucket — AWS S3, Cloudflare R2, MinIO, Backblaze B2, or anything speaking the S3 API.	Endpoint, region, bucket, access key ID, secret access key, optional prefix

Select Add destination, choose the type and give it a name you will recognise in a list.
For S3-compatible, enter the endpoint (HTTPS, public address, no credentials in the URL), region, bucket and keys. Keys are encrypted at rest and never shown again.
Select Save destination, then Test saved connection. Pallara writes, reads back and removes a small test object. Connection successful is the only state that lets an export run.

Take careAn export set will not run against a destination that has not passed a connection test — including after you change its endpoint, region, bucket or keys, which clears the previous result. If a scheduled run reports the destination is unusable, re-test it.

TipEditing a destination and leaving the credential fields blank keeps the saved keys. You only retype a key when you are actually replacing it.

2. Create an export set

An export set is what to send, and when. The wizard has four steps: Destination, Data, Schedule, Review.

Destination. Choose one that has passed its connection test.
Data. Start from a curated package — Learner lifecycle, Academic structure and delivery, Assessment and results, Attendance and engagement, Finance, Compliance and operations — or pick resources individually. Both end in the same place: a list of resources you can then trim.
Data, continued. For each resource, choose the columns you want and, optionally, a filter. Operators are equals, does not equal, greater than, greater than or equal, less than, less than or equal, is one of, and contains.
Schedule. Manual only, Hourly or Daily. For hourly and daily, set the exact local time and the IANA timezone — Pacific/Auckland by default. Every run time is worked out in that timezone, so a 2:30am daily export stays at 2:30am local rather than sliding by an hour twice a year.
Review. The preview shows the schema and an estimated row count per resource before anything is written. Select Create export set.

NoteOn the two days a year New Zealand changes its clocks, a schedule set inside the changed hour behaves the way the clock does. In September, when 2:00–3:00am does not exist, a 2:30am run happens at the first valid moment instead. In April, when that hour happens twice, it runs twice — two complete snapshots of the same day, under two run folders. If your loader keys on the calendar date rather than the run id, that is the day to check it on.

3. Run it, and check what arrived

Select Test export to run once, now, without waiting for the schedule.
The run appears in the history as Queued, then Running, then Succeeded, Partial failure or Failed.
Open the run for Export run details: every resource, its row count, its files, each file's size and checksum, and the reason for anything that failed.
Download a part file from the run to confirm the columns and values are what your pipeline expects.
Read manifest.json at the root of the run. It names every part, its row count and its SHA-256, plus the run's totals — that is what your loader should read first. It carries a schemaVersion (1.0 today); read it, and fail loudly on a version you were not written for rather than guessing at a changed shape.

From the card you can Pause schedule, Resume schedule, edit, retry a failed run, or delete the export set. A paused set keeps its history.

What the files look like

dw-exports/<export-set-id>/<run-id>/
  manifest.json
  students/part-000001.csv
  students/part-000002.csv
  enrolments/part-000001.csv
  programmes/part-000001.csv

CSV, UTF-8, RFC 4180 quoting. One header row per part file.
500 rows per part. The last part of a resource is whatever is left, and a resource with no rows still gets a header-only file — so a missing file means something went wrong, not that there was nothing to send.
Every run gets its own folder. Runs never overwrite each other, so a failed run cannot leave a half-written dataset where the previous good one was.
Dates that are a day are written as `YYYY-MM-DD` in New Zealand time — a date of birth, an enrolment start or end, a census or completion date, a programme term, an intake, an attendance date. Everything else is a full ISO-8601 instant. The manifest names which columns are which per resource, so your loader never has to guess. This matters: a date of birth read as an instant and truncated in UTC comes out a day early.
Checksums are per file. Compare the SHA-256 in the manifest against what you downloaded before you load it.

What is in a snapshot, and what is not

Complete, every time. A run sweeps a resource from end to end with cursor paging — there is no row ceiling, and no offset that could skip or repeat rows partway through.
Test learners are excluded, always.
Confidential learners are excluded. An export is a machine read, and a machine credential is not a member of staff.
Notes written for a narrower audience are excluded. A case note scoped to pastoral staff, to its author and managers, or to chosen roles never reaches a file.
Excluded columns stay excluded. Credentials, IRD and passport numbers, conviction fields, medical notes and learning accommodations are not exportable — the same allow-list the API uses.
Fifty of the fifty-one API resources are exportable. documents is the exception and does not appear in the picker at all: a document's owner is a polymorphic link that cannot yet be checked against a protected learner, so it is left out rather than exported unguarded. If you need document metadata in a warehouse, say so and it moves up the list.

Version 1 limits

Stated so nobody plans around something that is not there:

Full snapshots only. Every run sends the whole dataset. There is no incremental change feed and no deletion tombstone — your loader should replace, not merge.
CSV only. No Parquet, Avro or JSON output.
S3-compatible storage only. No native Snowflake, BigQuery, Redshift or Azure connector; load from the bucket instead.
No joins, no SQL, no transformations. One file set per resource, exactly as the columns are stored. Shape it in your warehouse.
Manual, hourly or daily. No arbitrary cron expression.
No public download URLs. Files go to your destination; run files are downloadable from the portal by staff who have access.

If one of these is on your critical path, tell us which — that is what sets the order they are built in.

Was this guide helpful?