Propagate Schema Updates¶

What it does¶

Automatically detects and propagates schema changes (new/deleted tables and columns) from a source app (e.g., Salesforce) to a destination app (e.g., Postgres). When a user adds or removes a field in the source, the corresponding column is created or removed in the destination.

Why it exists¶

Without this, users would need to manually edit and re-save their sync every time a field is added or removed in the source app.

How it works¶

Detection¶

A scheduled job polls the source app's schema at a configurable interval (default: 1 hour). It compares the stored schema against a freshly generated one from the external platform API.

Matching strategy: Matches by immutable ID first, then by name as fallback. This prevents renames from appearing as delete + create.

What gets detected:

Level	Change types
Tables	New, deleted, updated, unchanged
Columns	New, deleted, updated, unchanged

Flow¶

Scheduler picks up detect_schema_changes task
    |
    v
POST /detect-schema-changes/ endpoint
    |
    v
Fetch stored source schema from DB
    |
    v
Call connectors to generate fresh schema from external platform
    |
    v
Compare stored vs generated (detect_changes.py)
    |
    v
No structural changes? --> Update metadata only, return
    |
    v
Apply changes respecting propagation settings (apply_changes.py)
    |
    v
Create new tables/columns in destination via setup_apps_for_sync
    |
    v
Clean up deleted items from internal DB (cleanup.py)
    |
    v
Update base_schema in DB
    |
    v
Send email notification + Slack alert

Detection can also be triggered by errors¶

When a sync error occurs, an immediate detect_schema_changes ping is created (triggered_by_error=True). Configured per-app via create_detect_schema_changes_ping_on_each_error.

Configuration¶

App-level (`app_options`)¶

Option	Type	Description
`enable_detect_schema_changes`	bool	Master on/off toggle
`detect_schema_changes_frequency_in_seconds`	int	Poll interval
`propagate_table_creations_to_target_app`	bool	Auto-sync new tables to destination
`auto_unsync_deleted_tables`	bool	Auto-remove deleted tables
`excluded_external_table_names`	list	Table names to skip
`excluded_external_table_names_regexes`	list	Regex patterns to skip

Table-level (`table_options`)¶

Option	Type	Description
`propagate_column_creations_to_target_app`	bool	Auto-sync new columns to destination
`auto_unsync_deleted_columns`	bool	Auto-remove deleted columns
`excluded_external_column_names`	list	Column names to skip
`excluded_external_column_names_regexes`	list	Regex patterns to skip

Frequency options (UI)¶

Label	Seconds
Every 30 minutes	1800
Every 1 hour (default)	3600
Every 6 hours	21600
Every 12 hours	43200
Every 24 hours	86400

Scheduler¶

Stored in schedule_generic table:

Field	Value
`type`	`detect_schema_changes`
`concurrency_partition_id`	`dsc_{base_id}`
`custom_unique_constraint`	`dsc_{base_id}_{app_id}`
`max_concurrent_processes`	1

Only one detection can run per base at a time.

Error handling¶

Deleted elements with auto_unsync disabled¶

When deleted tables/columns are detected but auto_unsync is off: - Base is paused - Email notification sent to user - internal_disabled_detect_schema_changes = true set to prevent infinite loops - User must manually resolve and re-enable

Concurrency¶

Base status set to "processing" during propagation
Returns HTTP 200 even if already processing (prevents scheduler retries)

Ping lifecycle¶

When enable_detect_schema_changes is set to false, the ping is NOT deleted. A running ping may take up to 10 minutes — if deleted and user re-enables quickly, duplicate pings could run concurrently. The ping self-terminates when finished.

Notifications¶

Schema changes applied¶

HTML email summarizing: - New tables added (green) - Deleted tables removed (red) - Per-table column changes (new, deleted, renamed)

Rate limited: 1 notification per base per 60 seconds.

Error notifications¶

Sent when propagation fails. User needs to verify external apps and re-enable detection.

Key files¶

Component	Path
Main endpoint	`fe-logics/endpoints/common/detect_schema_changes/endpoint.py`
Change detection	`fe-logics/endpoints/common/detect_schema_changes/detect_changes.py`
Change application	`fe-logics/endpoints/common/detect_schema_changes/apply_changes.py`
Cleanup deleted items	`fe-logics/endpoints/common/detect_schema_changes/cleanup.py`
Utilities & notifications	`fe-logics/endpoints/common/detect_schema_changes/utils.py`
Settings validation	`fe-logics/validation/validate_schema_propagation_settings.py`
Frontend tab	`wf-fe/src/app/(carbon)/.../propagate-schema-updates/`
Error-triggered ping	`connectors/utils/detect_schema_changes.py`
Type definitions	`fe-logics/utils/types.py`