Media Sync

Table rows go to BigQuery with Supabase Pipelines. Scout RS copies Storage objects so those rows' file_path values exist in GCS at the same key.

Enable the optional media feature (keeps object_store out of default device builds).

What is copied

Both artifacts and events contribute file_path, thumbnail_file_path, and proxy_file_path, and each resumes on (updated_at, id) so derivatives written after the row was inserted are picked up on a later run.

Example: DB artifacts/42/clip.mp4 → gs://my-bucket/artifacts/42/clip.mp4.

Environment

SCOUT_DATABASE_REST_URL=https://<project>.supabase.co
SUPABASE_SERVICE_ROLE_KEY=...

Plus GCP Application Default Credentials (GOOGLE_APPLICATION_CREDENTIALS / GOOGLE_SERVICE_ACCOUNT).

The GCP principal needs storage.objects.get for destination checks and object creation permissions. roles/storage.objectUser includes both; if you currently use roles/storage.objectCreator, also grant roles/storage.objectViewer.

Add a lifecycle rule on the destination bucket to abort incomplete multipart uploads (7 days is typical). The sync aborts its own uploads on failure, but a hard kill of the process cannot, and GCS bills for orphaned parts:

gcloud storage buckets update gs://my-scout-media \
  --lifecycle-file=lifecycle.json

{
  "lifecycle": {
    "rule": [
      {
        "action": { "type": "AbortIncompleteMultipartUpload" },
        "condition": { "age": 7 }
      }
    ]
  }
}

CLI

cd scout_rs
cargo run --features media --bin sync_media -- \
  --gcs-bucket my-scout-media \
  --job-id nightly

One source:

cargo run --features media --bin sync_media -- \
  --source artifacts \
  --gcs-bucket my-scout-media \
  --job-id artifacts_media

Checkpoints are written to {checkpoint-dir}/{job_id}_{table}.json, so the same --job-id resumes correctly whether you run --source all or a single source. Event checkpoints written before events.updated_at existed hold an id-only cursor and are rejected; rerun those under a new --job-id.

Re-running the same job resumes after the last page whose objects were fully handled.

Throughput

--concurrency (default 4) sets how many objects are mirrored at once; each object additionally uploads up to 4 parts concurrently. Raising it trades memory for latency hiding, since every in-flight object holds up to ~25 MiB of upload buffers — at the default that is roughly 100 MiB, so size it against the memory limit of the container you run in.

Each object is checked in GCS before its Supabase body is opened. Existing destination keys are skipped, avoiding duplicate download and upload work after a checkpoint reset or overlapping job.

A page never checkpoints until all of its objects settle. When one object fails, the rest of the page still finishes rather than being cancelled, so no upload is dropped without aborting; the run then fails and the page is retried from the last checkpoint.

Missing source objects

Rows can reference Storage objects that no longer exist, usually because the object was removed while the row stayed behind. Those paths are skipped rather than treated as failures, so a handful of stale rows cannot block the rest of the sync or stall the checkpoint.

Each skip is logged at warn and counted in the run summary:

Mirrored 193 objects from 208 rows (1 pages) on artifacts; 4 already present, 15 missing
  missing: artifacts/42/clip.mp4

MediaSyncSummary::existing_objects and MediaSyncSummary::missing_paths expose both skip counts programmatically.

Only an absent object is skipped. Authentication, permission, and server errors still fail the run, as does a missing bucket — Storage reports both a missing object and a missing bucket as 404, and treating the latter as skippable would mark every object absent and advance the checkpoint past the whole table.

Library

use scout_rs::media_sync::{MediaSource, MediaSyncClient, MediaSyncOptions};

let mut client = MediaSyncClient::from_env("my-scout-media")?;

let summary = client
    .sync(MediaSyncOptions::new(
        MediaSource::Artifacts,
        "artifacts_media",
    ))
    .await?;

Limits

• Does not replicate table rows. Use Pipelines for BigQuery.
• events.media_url without a Storage file_path is not copied.
• Destination checks compare keys, not contents; replacing a Supabase object at an existing key does not overwrite the GCS copy.
• Missing source objects are skipped, so a warehouse row can exist without a GCS object.