QuantDinger Backend Concurrency Model¶

This document defines how the backend should behave under concurrent users, strategies, workers, API calls, and external providers.

Core Principle¶

All operations that can mutate trading, payment, job, or account state must be idempotent or explicitly serialized. Retrying a request must never create an unintended duplicate order, duplicate membership activation, duplicate job, or conflicting position row.

Concurrency Domains¶

Domain	Parallelism Allowed	Serialization Key	Required Protection
Strategy lifecycle	Different strategies may run in parallel	`strategy_id`	start/stop lock, stale status repair
Strategy symbol execution	Different symbols may run in parallel	`strategy_id:symbol:side`	execution lock, position reconciliation
Quick trade order placement	Different users/symbols may run in parallel	`user_id:credential_id:market:symbol:side`	idempotency key, client order id
Pending order dispatch	Multiple pending orders may run in parallel	`pending_order_id` and venue order id	claim-before-dispatch, retry dedupe
Grid resting orders	Different grid cells may run in parallel	`strategy_id:symbol:cell_index`	DB unique cell, fill reconciliation
Account mirror	One row per credential/market/instrument/side	`credential_id:market_type:inst_id:side`	upsert, no blind delete on partial data
Backtest jobs	Many jobs may run in parallel	`job_id`	bounded worker pool, no shared mutable state
Agent jobs	Many jobs may run in parallel	token + kind + idempotency key	DB unique index, replay returns same job
USDT payments	Many users may pay in parallel	`order_id` and tx hash	order status state machine, one activation
Market data cache	Reads parallel, refresh single-flight per key	cache key	stale-while-revalidate, TTL
SSE streams	Many read streams allowed	`job_id:user_id`	heartbeat, disconnect handling

Existing Protections To Preserve¶

Agent jobs have job_id and idempotency_key columns with a unique index.
Agent quick-trade and backtest routes already use idempotency helpers.
Strategy positions and account mirror tables use uniqueness on natural keys.
Grid cells use a unique (strategy_id, symbol, cell_index) constraint.
Data providers use in-process single-flight locks for cache refresh.
USDT orders use uniqueness to avoid active amount/address collisions.

Gaps To Audit During Refactor¶

Worker startup under Gunicorn multi-worker mode: only one process should own singleton workers unless a distributed lock is introduced.
In-process locks are not enough when the app runs in multiple processes or hosts.
Quick-trade human web routes need the same idempotency posture as Agent routes.
Strategy restore must not start duplicate execution loops after a crash/redeploy.
Pending-order dispatch should atomically claim work before calling exchanges.
Exchange retry code must dedupe by client_order_id where the venue supports it.
Long-running LLM/backtest endpoints should avoid blocking request threads.

Required Patterns¶

Idempotency Key¶

Use for any user-triggered operation that may be retried by browser, mobile app, agent, proxy, or worker:

order placement
close position
backtest submit
USDT order create/confirm
strategy start/stop when triggered by API

Recommended key shape:

actor_id:operation:target:client_request_id

If the client does not supply a request id, generate a server-side key from stable input only when duplicate replay is safe.

Execution Lock¶

Use for operations that must not overlap:

strategy:{strategy_id}:lifecycle
strategy:{strategy_id}:symbol:{symbol}:side:{side}
credential:{credential_id}:instrument:{inst_id}:order

In-process threading.Lock is acceptable only for single-process local mode. Production-safe locks must use database row locks, advisory locks, or Redis locks.

Claim Before Work¶

Workers must claim a row before external side effects:

Select eligible row.
Atomically transition from pending to processing.
Commit.
Call external exchange/payment/LLM.
Store result and final status.

No worker should call an exchange before it owns the row.

State Machine¶

Order-like records should have explicit states. Avoid free-form status strings.

Suggested lifecycle:

pending -> processing -> submitted -> filled
                         -> partially_filled
                         -> failed
                         -> cancelled
                         -> expired

Payment lifecycle:

pending -> seen_on_chain -> paid -> confirmed -> fulfilled
                    -> expired
                    -> failed

Rate Limits And Backpressure¶

External market data providers must have per-provider timeout and retry policy.
Exchange adapters must avoid unbounded retry loops.
LLM calls should run through job workers for long outputs or streaming.
SSE streams must send heartbeat events and release resources on disconnect.
ThreadPoolExecutor sizes must be bounded and documented.

Test Requirements¶

Before changing high-risk concurrency code, add or update tests for:

duplicate API request returns existing result
two concurrent starts do not create duplicate strategy loops
two concurrent order attempts do not place duplicate venue orders
pending order worker can retry after failure without duplicate fill records
USDT confirmation can run twice and activate membership once
agent job idempotent replay returns the same job_id

Deployment Notes¶

Local single-process mode may rely on in-process locks. Docker/Gunicorn production mode must not assume process-local state is globally unique. Any worker that can create external side effects must have a single owner or a distributed lock.