Bulk Transaction Conversion & Retry Queue

A design reference for converting many source records into a related target record type in one background run, with per-item logging and retry

1. Requirements

1.1 Functional requirements

• Let a person select many submitted source records on a list screen and convert all of them into a related target record type in one action (for example, many quotations into orders, or many orders into invoices), instead of converting each one individually.
• Which target types are reachable from a given source type is a fixed, small matrix of (source type, target type) pairs, each bound to its own conversion function — not a generic “create any related record” mechanism. Another part of the system can extend the matrix through a discovered extension point.
• Exclude source records whose status makes conversion nonsensical (on hold, or closed) before anything is attempted, and report what was excluded.
• Log every attempted conversion, success or failure, as its own row: source record, source and target types, timestamp, outcome, and — on failure — the full error detail.
• Let a failed conversion be retried without re-selecting the original batch; retry works directly off the failure log.
• Run large batches as a background job so the triggering request returns immediately.

1.2 Non-functional requirements

• Failure isolation: one record’s failure must not abort the rest of the batch or leave a half-written target record behind.
• Retry safety: retry only reconsiders rows already logged failed and not yet retried; never a row that already succeeded.
• Auditable failure: a failed row keeps the full error text, not just a flag.
• No dedicated infrastructure: nothing beyond the fixed matrix — no template setup, no separate queue provisioning.

1.3 Constraints

• The matrix is generic, shared infrastructure behind many unrelated conversions (order-to-invoice, quotation-to-order, order/invoice-to-settlement, and others), not built for one pair.
• Only already-submitted records are offered for selection; drafts are filtered out before a batch is created.
• Retry reconsiders logged failures only; it has no notion of “the original selection” once a batch has run.

2. High-Level Design

2.1 Component diagram

2.2 Data flow — one batch run

1. Selection. A person checks several submitted records and picks a target type; drafts are excluded up front.
2. Intake filter. Records “On Hold” or “Closed” are pulled out before anything else runs and reported back — these never get a log row and are invisible to every later step, including retry.
3. Enqueue. If anything remains, a background job is queued.
4. Per item: open a savepoint, resolve the conversion function for (source type, target type), and call it. On success, insert the resulting target record unsubmitted — standard validation and mandatory-field checks are deliberately bypassed, since a person is expected to review the draft before submitting it — and log a success row. On failure, roll back to the savepoint and log a failure row with the full error text.
5. Summary popup: fully successful, partially successful (linking to the log), or fully failed.

3. Deep Dive

3.1 Data model

Conversion Run Summary — a virtual, read-only record keyed by calendar date, with no table of its own: every read computes succeeded/failed counts live from Conversion Attempt Records for that date. Unrelated runs on the same day roll into one summary row that cannot distinguish which run produced which count.

Conversion Attempt Record — the real, persisted row: a dynamically-typed reference to the source record, source and target types, a date and time, an outcome of Success or Failed, an error-detail field populated only on failure, and a retried marker (0 or 1). Every first-time attempt appends a new row, including a second, independent batch touching a source record already converted once — nothing checks for that.

Target draft record — whatever the matrix’s conversion function produces, always inserted unsubmitted. Where the target is a settlement voucher (a draft accounting posting for money to be received or paid), the function additionally receives the source type explicitly, since settlement-voucher creation is shared logic serving many source types; every other pairing’s function is already bound to one source type and needs only the record’s name.

3.2 The conversion matrix

Fixed pairs, by source type: Sales Order (to Sales Invoice, Delivery Note, or a settlement voucher), Sales Invoice (to Delivery Note or a settlement voucher), Delivery Note (to Sales Invoice or a Packing Slip), Quotation (to Sales Order or Sales Invoice), Supplier Quotation (to Purchase Order or Purchase Invoice), Purchase Order (to Purchase Invoice, Purchase Receipt, or a settlement voucher), Purchase Invoice (to Purchase Receipt or a settlement voucher), Purchase Receipt (to Purchase Invoice) — a literal table of function references, not one generic rule. A separate part of the system can extend it through a discovered extension point, merged over the built-in table once per run, without touching this module’s own code.

3.3 The retry model

Retry triggers two ways: manually, from a button on one date’s Conversion Run Summary, or via a periodic, unscoped sweep defaulting to today. Both call the same routine.

• Candidates: Conversion Attempt Records with outcome Failed and retried = 0 for the target date. A row already at retried = 1 is never selected again by either trigger.
• Re-attempt: the same savepoint-guarded call into the same conversion function runs again, driven purely by what the failed row recorded — not a fresh list.
• Outcome: either way, the row’s retried flag is set to 1 and its outcome/error text are updated in place — the one path in this module that updates a log row instead of appending one. A row that fails again (Failed, retried 1) is terminal: nothing here revisits it without a person intervening, or an unrelated fresh batch.
• Not checked: neither the original batch nor retry verifies a source record wasn’t already converted successfully. Whether a repeat attempt succeeds, fails, or produces a second target draft depends entirely on the target-side conversion function’s own validation — outside this module (see the order-to-cash lifecycle, documented separately, for what several of these pairs actually enforce).

3.4 Error handling

A savepoint wraps each attempt individually, so one exception cannot roll back or block any other item in the run — this is what makes partial failure resumable: a batch with a handful of failures leaves every successful draft in place, with only the failed subset needing revisit. Records excluded by the intake filter never generate a log row, so they sit outside retry’s reach — reconversion needs re-selecting them once status changes.

4. Scale and Reliability

• Load pattern: bursty and person-triggered, plus one low-frequency, unscoped periodic sweep sharing the same maintenance cadence as several other periodic checks in the system.
• No per-item parallelism: a batch is one background job processing serially, not sharded across workers, however large.
• Idempotent scheduling: the sweep and the manual retry button both select on retried = 0, so either can run without conflicting with the other.
• Monitoring gap: nothing alerts on accumulating failures; a row that survives one retry cycle (retried=1, still Failed) is easy to lose track of without someone checking a date’s summary.

5. Trade-off Analysis

6. What to Revisit as the System Grows

• Duplicate-conversion guarding inside the queue itself, closing the gap in §3.3/§5 without relying on each of the eight target-side functions independently.
• Alerting on stuck failures — surfacing rows sitting at Failed/retried=1 rather than leaving them to be found by chance.
• A lookback window on the periodic sweep, instead of only “today,” so a failure crossing a date boundary unretried is not permanently outside its reach.