Skip to content

Variables

Max concurrent processes per concurrency partition

  • Variable name: max_concurrent_processes_per_concurrency_partition
  • What it controls: Upper bound of concurrently running processes within a single concurrency partition (per base+app or partition group). The generic scheduler uses this to decide how many tasks can start in parallel for a given partition.

Where this value comes from

  • It is computed from the app's rate limit in code, not a static env var.
  • Source helper (single source of truth):
  • Python: stacksync_utils/configurations/max_concurrent_processes_per_concurrency_partition.py
    • calculate_max_concurrent_processes_per_concurrency_partition(rate_limit, frequency)
    • calculate_max_concurrent_processes_per_concurrency_partition_from_app_schema(app_schema)
  • Formula: floor division with safety buffer
  • max(1, rate_limit // (2 * frequency))
  • Example: rate=300000, frequency=86400 → 300000 // (2*86400) = 1

How it’s used

  • Persisted in DB on each scheduler row creation/update, in schedule_generic.max_concurrent_processes_per_concurrency_partition.
  • The generic scheduler enforces it when selecting runnable tasks per concurrency_partition_id.
  • Set/updated by:
  • Connectors when creating read pings/read-full pings/rescrape tasks (e.g. HubSpot, Salesforce, NetSuite, etc.).
  • Frontend-Logics when updating an app’s rate limit; it recomputes and updates existing pings for that partition (with a minimum of 2 in some places when persisting to DB).

How to use it in code (DO this)

  • Always derive it from the app schema with the centralized helper:
  • calculate_max_concurrent_processes_per_concurrency_partition_from_app_schema(app_schema)
  • Avoid ad-hoc calculations like rate/2 or re-implementing the formula.
  • If a component requires a minimum when persisting (e.g., GREATEST(2, …) in scheduler rows), apply that minimum at the persistence boundary, not in the core helper.

Key implementation references

  • Helper functions (calculation):

  • python-utils-3/stacksync_utils/configurations/max_concurrent_processes_per_concurrency_partition.py

  • Places that compute/use it during process creation:

  • Connectors (examples):
    • connectors/apps/*/switch_app.py when creating read pings/read full processes
    • connectors/commons/read_full.py when partitioning read-full
    • connectors/utils/rescrape.py when upserting rescrape tasks
  • Event-forward helpers:
    • event-forward/utils/rescrape.py, trigger forwarders (Zoho CRM, Attio)
  • Frontend-Logics (rate-limit edit path):
    • frontend-logics/endpoints/common/bases.py recomputes and updates existing rows

Notes

  • There is a related (different) knob: max_concurrent_processes_per_connection for connection-scoped concurrency.
  • Some connectors or schedulers may enforce >= 2 on persisted rows to avoid starvation; the helper itself returns >= 1 by design.