Parallelization: sectioning and voting

Why it matters

A chain is only as fast as the sum of its steps. When the steps do not depend on each other, that sum is waste: reviewing twelve clauses of a contract one after another takes twelve calls of time when it could take one. And a single call is only as reliable as one sample from a probabilistic model; for a safety decision that must be right, one opinion is not enough. Parallelization answers both. Sectioning cuts latency on tasks that split; voting buys confidence on tasks that do not. Both pay in tokens. Interviewers ask about it because latency ceilings and reliability targets are the constraints every real system has, and the pattern makes the cost of meeting them explicit.

The idea

Two shapes share one mechanism: issue several model calls at the same time, wait for them, aggregate.

Sectioning

Split the task into parts that do not need each other's output, run one call per part, merge. A document review becomes one call per section; a batch of tickets becomes one call per ticket; a question that needs a legal view and a financial view becomes two calls with different prompts over the same input. Latency drops from the sum of the calls to the slowest one, plus the merge. Tokens stay roughly level with the sequential version, because the same work is done, with a small overhead for repeating shared context in every call. The hard part is the split: parts must be independent, and the merge must handle what the split lost. A clause that contradicts another clause is invisible to a reviewer who saw only one of them, so the merge step reads all section outputs together, or a final cross-cutting call does.

Voting

Run the same task several times, with the same prompt at non-zero temperature, with different prompts, or on different models, and aggregate the answers. Three judges read a piece of content and each says safe or unsafe. Five calls grade a translation and the median score stands. The aggregation rule follows the goal. Majority suits classification where errors are random. Strictest wins suits safety: one unsafe vote blocks. Any pass suits generation: produce three candidates and let a program check pick the first that compiles. Voting costs n times the tokens of a single call and only slightly more latency. It helps when the single-call error is random across runs; it does nothing against an error every run shares, such as a misread instruction, and diverse prompts or models exist for exactly that.

Partial failure

Parallel calls fail independently. Rate limits, timeouts and malformed outputs will take out one of three on a bad day. Decide in advance. For sectioning, a missing section is reported as missing, never silently dropped from the merge. For voting, set a quorum, so two valid votes of three can still decide while one of three cannot, and the item is retried or escalated. Give every call its own timeout shorter than the ceiling, and a total deadline for the fan-out.

results = await gather(*(judge(content) for judge in JUDGES), return_exceptions=True)
votes = [r for r in results if not isinstance(r, Exception)]
if len(votes) < QUORUM:
    raise PartialFailure(len(votes))            # retry or escalate, never guess
decision = "unsafe" if "unsafe" in votes else "safe"   # strictest wins

When parallel beats sequential

Three triggers. The subtasks are independent, so nothing is gained by ordering them. There is a latency ceiling the sequential version breaks: twelve sections at four seconds is 48 seconds in sequence and about five in parallel. Or the task benefits from several views: judges with different instructions, models with different blind spots. Absent all three, sequential is simpler and cheaper to reason about.