P2P downloader architecture and mechanisms

01The problems a downloader really needs to solve

Add hundreds of downloads at once: some start quickly, some never find a source, and others show many connected peers but barely make progress. Raising the concurrent-download limit sometimes helps; at other times it only increases disk activity and memory use, or even slows downloads that were working normally.

The reason is that a downloader handles several distinct questions at once: What content am I looking for? Who has the parts I need? Can they transfer them now? Can the local machine keep up? Is the received data correct? These activities differ in speed and cost, and do not all finish in a neat sequence.

This article works down from the overall architecture of a proposed downloader supporting multiple P2P protocols. BitTorrent and eD2k/Kad illustrate actual protocol behavior; the nine components, dual loops and dynamic download pool are one architectural choice for organizing those capabilities.

First, distinguish the forms data takes in the system

In BitTorrent, for example, peers exchange information about the pieces they hold, and the receiver then requests data. Received pieces still need verification against the metadata. This lets a downloader obtain data from multiple peers at once, while making connectivity, traffic and correct new data three separate things to observe.^[01]

02Overall architecture: how nine components work together

Start with three questions to understand this architecture: Where do control decisions go, where does payload go, and where is state stored? Pressing pause only needs to change a task's execution intent. The protocol engine continues to handle the gigabytes being downloaded; those bytes do not need to pass through the management interface or database.

Logical responsibilities and process placement are separate decisions

The nine names represent nine responsibilities, not necessarily nine services from day one. One Rust daemon can host the coordinating logic of Manager, Search, Scheduler, Connection Manager and Store. Native engines can then live in the same process or in separate Component hosts according to their embedding capabilities, implementation languages and fault-isolation needs.

In-process calls and data exchange are more direct, and deployment is simpler. Separate processes isolate engine crashes, native libraries and resource limits, at the cost of more complex communication, restarts and state synchronization. Those costs—not the number of boxes in an architecture diagram—should determine whether processes are separated.

A download task does not need its own engine process either. Multiple tasks can usually share an engine runtime, connection management and source-discovery facilities. More tasks primarily add state and the resources actually needed, rather than multiplying the entire system.

03How Component connects different protocols

A common mistake in a multi-protocol downloader is reducing every protocol to the same start, pause and progress-percentage interface. Operations appear unified, but crucial information disappears: BitTorrent choking, remote eD2k queues, source reachability, and whether an engine can actually close just one connection.

Component preserves these differences while letting upper layers manage them through common questions: What data is available now? What resources will transfer require? What would be lost by stopping? Is the observed wait caused remotely or locally?

Put each protocol capability in the right place

Three decision layers, each answering a different question

Scheduler decides which tasks receive how many resources; Connection Manager decides which sources or connections are suitable now; Engine decides which piece to request from which peer next. This lets the outer layer manage the whole host's load while the engine handles individual requests according to live protocol state.

For example, Scheduler may allow task A to keep downloading while Connection Manager temporarily avoids one source that repeatedly times out. Engine can still request pieces from other sources. If the user has paused task A, no layer should resume transfer merely because a faster source was found.

Visibility does not imply equally precise control

An external engine's management API may expose only torrent-level summaries, while its underlying library provides per-peer piece and request information. These capabilities must not be conflated. Likewise, banning an IP and closing only the current connection have different scopes; the adapter must preserve that distinction.

qBittorrent provides a concrete example: in the pinned source revision cited here, banPeersAction() parses endpoints and then calls banIP(). That action therefore cannot be presented as a control affecting only one connection. This example concerns the cited revision and illustrates differences in adapter capabilities.^[18]

With incomplete information, an upper layer can still perform coarser task scheduling, but it cannot infer unobserved details. Showing 'not provided by the engine' is more useful than displaying fictitious zero traffic or falsely precise controls.

04Finding content and finding someone who can send it

Before discovering sources, establish that everyone means the same content. A filename is only display information: identical names may denote different files, and different names may denote identical data. Multi-protocol systems face another issue: hashes in different protocols may use different algorithms, segmentation schemes, or even hash different kinds of objects.

An infohash identifies a torrent's content description

A BitTorrent v1 infohash is the SHA-1 hash of the original bencoding of the info dictionary. This dictionary records the content structure and piece-verification information; the infohash is its digest, not a SHA-1 hash of the entire downloaded file. In v2, the infohash uses SHA-256, and a per-file Merkle tree organizes data verification.^[01][03]

Consequently, byte-identical files may belong to different torrents if their torrent descriptions differ. Selecting only two files in a multi-file torrent is another matter: the torrent's identity remains unchanged, but the ranges still needed by that task change.

The system should store content identity, selection scope and network scope separately. BitTorrent, eD2k and other identifiers must not be merged just because their strings look similar. Cross-protocol data reuse also requires verification of content correspondence, piece boundaries and verification methods.

Why do magnet links often wait at 'fetching metadata'?

A magnet link may contain only a content identifier and a few location hints, without a complete file structure. The client first finds peers, then uses metadata exchange to obtain the info dictionary and check it against the known infohash. BEP 9 transfers this part, not all top-level fields of the complete .torrent file; v2 verification also needs its corresponding hash information.^[02][03]

It is reasonable for the interface to show no file progress yet: the system is obtaining information needed for subsequent requests. If no usable peer exists, increasing the transfer budget cannot conjure up metadata.

Different discovery mechanisms operate on different indexes

The upper-level Search component can therefore support both finding content by keyword and finding sources for a known identifier, without assuming every protocol has the same search capabilities. Extensions to other networks must likewise preserve their own indexing, queuing and identity models.

Private torrents have their own source boundaries

When metainfo explicitly sets private=1, BEP 27 requires operation within the private tracker's source scope. DHT, PEX and similar mechanisms must not spread sources as they would for an ordinary public torrent.^[06]

Before metadata is available, not yet knowing whether a torrent is private does not justify arbitrarily adding public trackers. This architecture retains original sources together with discovery policies, preventing a global desire for more sources from crossing an individual task's privacy boundary.

05Why large download workloads need two loops

Suppose 1,000 targets are added at once. If only tasks occupying download slots can discover sources, a few unresponsive tasks at the front can hold back the rest. The opposite extreme—immediately creating many connections for all 1,000 targets—can exhaust local resources first.

The dual-loop design separates these activities: the discovery loop maintains source information; the transfer loop fetches actual data according to current evidence and resources. Every accepted target remains eligible for discovery scheduling, while the queries and connections actually issued concurrently remain resource-limited.

If target 900 returns first, handle target 900 first

Discovery need not wait for an entire batch. While target 1's tracker is still unresponsive, target 900 may already have found a peer holding missing pieces and can begin a trial. Each result updates independently; there is no requirement to wait until everyone responds.

This does not mean repeatedly querying every source. For identical content under compatible network scopes and policies, duplicate source queries can be combined so that different file selections share results. Tracker transports, DHT routing tables and eD2k server sessions also have shareable parts; metadata already being fetched usually does not need a duplicate job.

Retain leads so connections can be released when appropriate

Discovery can retain source addresses, advertised ranges, current service states and observation times. These leads survive the end of a brief connection and guide later retries.

Source information ages, however. A peer reachable five minutes ago may now be offline, and previously missing pieces may already have arrived elsewhere. Stale information can guide whom to query first, but is not proof of immediate download availability. An expired snapshot affects source assessment only; it does not invalidate locally verified data.

Discovery has its own cadence

Protocols themselves constrain requery frequency. Tracker response intervals, PEX update cadence and eD2k reasking mechanisms all affect the next reasonable query time. Global scheduling can prioritize queries and merge duplicate work, but rapid retries cannot eliminate a remote waiting policy.^[01][05][12]

Separating discovery from transfer lets a busy system keep finding new opportunities, while tasks with temporarily scarce sources need not occupy full transfer resources indefinitely.

06Brief probes and the real cost of connections

A brief probe before committing to transfer can reveal whether a peer is reachable, responds for the correct content, advertises particular pieces, or currently requires queuing. This helps choose the next source worth trying.

A probe answers only limited questions. A quickly established TCP connection proves only that this connection establishment was quick—not the peer's upload bandwidth, service stability or data integrity. Those require actual requests for useful data.

A probe is not the same as a half-open connection

In libtorrent, connecting denotes the half-open state while a connection is still being established; handshake indicates an open connection waiting for the protocol handshake. Obtaining a bitfield normally goes beyond connection establishment alone.^[09]

This article therefore calls the activity a brief probe session. Deliberately leaving TCP incomplete neither obtains the required protocol information nor eliminates resource use. For UDP trackers, DHT and mechanisms with shared transports, one logical probe must not be counted automatically as one independent TCP socket.

Limit concurrent population and creation rate separately

One limit governs the concurrent population: how many file descriptors, how much memory and how much engine state the currently live probes occupy. Another governs the creation rate: even if old probes finish quickly, that does not justify creating many new connections to one service within a second.

A file descriptor (FD) is an operating-system handle for resources such as open files and sockets; available FDs are only one constraint. A token bucket converts a permitted creation rate into consumable tokens and limits bursts. Used together, these controls constrain both ongoing cost and short-term impact.

Actual costs vary with protocol and response size. Beyond estimates, monitor connection timeouts, memory growth, event-processing delay and allocation failures. NAT tracking-table and ephemeral-port capacity may not be directly observable; if failure rates rise, conservatively reducing creation is more reliable than assuming unlimited capacity.

When a probe ends, release only resources that can be released

A temporary connection may be closed once source information is obtained and no native state requires it to remain open. Shared server sessions, DHT routing state and valuable queue eligibility must be handled according to the protocol, not torn down after every probe.

In eD2k in particular, queue state does not correspond one-to-one with a persistent TCP connection; reasks, queue ratings and related interactions may also use UDP. Whether a connection must remain open and when to reask should follow the native engine's mechanisms, not blanket claims that disconnection always loses one's place or has no cost at all.^[12]

07Why can a download remain incomplete despite many sources?

A source count tells us only how many candidates exist, not whether the complete content can be obtained. The useful questions are: Which pieces are still missing locally? Can the current sources collectively provide them?

A complete seed holds every piece, usually an obvious positive sign. Without one complete source, several peers holding complementary portions may still suffice. Conversely, hundreds of sources all missing the same final piece can leave progress stalled.^[12]

Two ratios distinguish what is present from what is still obtainable

Let S denote the data ranges required by this task, V the locally verified ranges, and Aⱼ the ranges source j recently advertised as available. These sets use compatible native verification ranges; multi-file content also requires attention to piece boundaries.

Here, bytes() counts bytes in a range. The example uses equal-sized pieces, so counting cells works directly; a real file's final piece may be shorter, making simple piece counts misleading. When M is empty, all required data has passed verification and can proceed to persistence and final publication; there is no need to divide by zero.

The first ratio asks how much of the requirement is covered by locally verified data together with sources' current advertisements. The second considers only data still missing. Neither turns a peer's advertisement into verified progress. If a protocol does not expose a piece list, leave that state unknown and use actual transfer to inform the assessment.

Rare pieces sometimes matter more than current speed

Even if source C in the example is slower, it holds pieces 10–12 that the others lack. Ranking only by MiB/s and replacing it with a fast source holding only pieces 4–9 may improve short-term traffic while reducing the chance of completing the file.

That is why global scheduling must preserve opportunities to obtain rare data. Engine handles piece selection within a download; Scheduler manages resources across tasks, preventing tasks with rare sources from being completely crowded out by higher-throughput tasks. libtorrent's piece picker considers rarity and priority, but global fairness is a separate concern.^[10]

08How peer data becomes a usable file

The actual data path begins once sources are available. Engine uses known pieces, outstanding requests and native service state to decide what to request from whom next. The outer download pool sets resource bounds; these detailed choices stay inside the engine.

A piece is a verification unit; a block is a request unit

In BitTorrent v1, metadata contains hashes for individual pieces. During transfer, a piece may be divided into multiple smaller block requests. A received block initially establishes only its received range; normally the whole piece must be assembled before its hash can be checked.^[01]

This does not mean the entire piece must remain in memory. The engine can write to a partial file first, then read it or reuse available data for verification. Receiving, writing and verification may interleave; 'received' alone does not mean data is safe to reuse.

Why keep multiple requests outstanding?

Sending one request and waiting for its response before sending the next creates idle gaps from network round trips. Pipelining keeps multiple requests in flight, so later requests are already being processed while earlier data returns. The BitTorrent specification explicitly describes this approach.^[01]

Too small a request window may leave bandwidth idle while waiting; too large a window accumulates buffers and slows cancellation or reallocation. When the network is not saturated, inspect existing sources' request pipelines before deciding to add more tasks.

Verification granularity also affects recovery costs

BitTorrent v2 organizes each file's hashes with a SHA-256 Merkle tree, relating data to an expected root through the tree. When a trusted root and recovery data are available, *Mule's AICH can locate smaller damaged blocks within a part, preserve correct data and reduce redownloading. Their formats and trust sources differ; they are not directly interchangeable hash trees.^[03][13]

A matching hash proves correspondence with the expected identifier. If the selected content was untrustworthy in the first place, downloading it correctly does not automatically make it a safe program or file.

Sometimes duplicate data helps a download finish sooner

Near the end of a download, only a few blocks remain, so any slow source can delay completion. BitTorrent endgame mode may request remaining data from multiple sources and cancel the other requests after one copy arrives. Cancellations and data responses can cross in transit, producing late duplicate blocks.^[01][07]

Measurements can therefore distinguish network-received bytes, nonduplicate required data and verified progress. Duplicates have a cost but may shorten the final wait. The duplication ratio alone does not establish that a source is deliberately wasting bandwidth.

09Trials: judge service using data the task actually needs

Discovery supplies leads; a trial reveals actual service. It fetches ranges genuinely missing from the task. Correct data remains useful for the subsequent download, avoiding a separate speed test that fetches unrelated content.

A trial is not intended to produce an eternal speed ranking. It asks whether this combination of sources, under current network conditions, local load and protocol state, can keep providing new data worth the resources invested.

Connecting, receiving service and receiving data are distinct observations

Some sources handshake quickly but remain queued; others take longer to start yet transfer steadily afterward. Measuring every no-data interval from connection establishment conflates native waiting with genuine request timeouts.

Service eligibility is not a remote bandwidth guarantee; it only means the conditions for making a particular request are currently met. Ordinary BitTorrent choking restricts transfers, but after Fast Extension negotiation, allowed-fast may permit requests for particular pieces while choked, and those requests may still be explicitly rejected. Eligibility therefore depends on the specific request and negotiated mechanisms.^[07]

Retain uncertainty when samples are insufficient

If a trial's resource window cannot obtain a complete verification unit, its outcome may be 'awaiting verification', not corrupt data. A high-RTT source that has only just begun service should not be judged conclusively from one short window either. Consider new data, verification progress, latency, duplication and local interference together, then decide whether further observation is worth its cost.

A small file may finish during the trial; proceed directly to verification and persistence. A slow source holding rare pieces may also deserve continued service. Its value cannot be reduced to a fixed MiB/s threshold.

Task states explain what is being waited for

These phases and source quality are separate dimensions. An incomplete source actively providing rare pieces is perfectly reasonable, as is a paused download whose completed local pieces remain valid. Compressing them into one good-or-bad score loses explanatory power.

10How the download pool expands, contracts and rotates fairly

A dynamic download pool is a continuously observing feedback controller: allocate resources to some tasks, measure their progress and pressure, then decide what to do next. The number of active transfers is an outcome of that process.

Inside Scheduler, responsibilities can be divided into three parts: PoolSupervisor summarizes the current situation, a capacity controller decides how much to increase or reduce overall investment, and a rotation mechanism selects suitable candidates and tasks to yield. These names describe internal responsibilities within the same global scheduling layer.

Resources form a set of constraints, not one remaining-download count

Five tasks may all be remotely limited and use almost no bandwidth; another may already saturate one disk's random-write capacity. Scheduling must therefore consider network input and output, pending writes, memory, verification work, CPU and FDs together, while preserving headroom for control operations.

This is a framework for resource allocation, not a claim that every cost can be accurately predicted. Some resources can be limited directly; others require adjustment based on live pressure. Work already dispatched whose reduction is unconfirmed may still be transferring, so its resources cannot yet be treated as fully available.

How backpressure propagates from disk to network

When data arrives faster than the disk can write it, pending-write buffers grow. Continued growth consumes memory and may delay verification and event handling. Adding peers then imposes more unprocessed data on the downstream path.

Backpressure tells the upstream side that the downstream side cannot keep up: reduce new data requests or transfer budgets first, allowing in-flight data and write queues to drain. Useful connections may be retained according to cost; there is no need to disconnect and restart the entire pool.

Pressure also has a scope. Tasks writing to a slow HDD need throttling, while those writing to a separate idle SSD may be unaffected. Linux PSI supplies signals of pressure from CPU, memory and I/O stalls, but is not a per-disk throughput gauge or a measurement of available Internet bandwidth.^[16]

Compare useful throughput at the same data layer

Network-interface traffic includes protocol overhead, retransmissions and traffic from other programs. Payload received by the downloader may still include duplicates or data that eventually fails verification. For long-term benefit, measure newly verified required bytes per observation interval; use deduplicated received progress as a short-term supplement.

Both measurements serve a purpose. Verified volume is closer to reliable progress, but grows in steps because of verification units and scheduling. Deduplicated received volume responds faster but remains unverified. Trends across several windows with comparable load are more stable than decisions based on a one-second peak.

Avoid oscillations caused by the controller itself

New tasks need time to establish connections, queue and fill request pipelines. If they are stopped immediately for showing no benefit, then restarted as soon as capacity looks idle, the system continually pays startup costs.

Smoothed observations, a settling period after expansion and different expansion/contraction thresholds can reduce this oscillation. Different thresholds create hysteresis: small fluctuations near a boundary do not trigger a reversal every time. Ordinary adjustments can wait for a clear trend; protective events such as a full disk or engine error should be handled immediately.

Fairness requires explicit opportunities

Keeping only today's fastest tasks gives untried tasks no chance to prove themselves; rare content may remain permanently behind popular content. The pool can reserve a small share of resources for unknown sources, long-waiting tasks and rare pieces.

Rotation also has costs: existing sources have established stable pipelines, while new ones may need to queue again. A reasonable approach is to compare candidates through limited trials, then replace lower-yield tasks that can yield cheaply. Fairness does not require restarting every download periodically.

11How Connection Manager understands each connection

The download pool sees aggregate resources and task benefit; Connection Manager focuses on individual connections. They inform each other: global scheduling needs to know which resources are valuable, while connection analysis needs to know whether local throttling, disk behavior or other tasks are interfering.

This component primarily puts scattered observations back into context. Before interpreting a zero-traffic connection, establish at least its role, request state, advertised data and whether the local machine is healthy.

Identify the role before judging payload traffic

Trackers, DHT nodes, eD2k servers and payload peers perform different jobs. An index service with no payload for a long time may be entirely normal; an upload-only connection cannot be judged by receive-side criteria either. Role classification is especially important when discovery and transfer coexist.^[04][08][19]

The same IP does not mean the same connection or person

Connections are recreated, engines restart, and one endpoint may serve multiple tasks. Tracking must identify the execution generation and actual connection to which an observation belongs. Otherwise, a delayed request to close an old connection could affect a newly established one.

Peer IDs, eD2k user hashes, endpoint addresses and local connection IDs serve different purposes. They can correlate protocol interactions, but are not authenticated real-person identities. Shared NAT, dynamic addresses and reconnection also make long-term ratings attached to a single IP imprecise.

eD2k High ID and Low ID are another distinction: they mainly reflect inbound reachability and whether callbacks are needed to establish connections. A higher number does not imply greater trust or speed, and Low ID does not mean servers relay all payload. Reachability affects source opportunities, but actual service must still be observed.^[14]

Unknown fields should remain explicitly unknown

Some engines expose per-peer piece lists, pending-write bytes, request timeouts and choke state; some management APIs show only aggregate speed. libtorrent's native peer_info contains several such observations, but whether an adapter exposes them depends on its actual interface.^[09]

Missing data may mean not yet observed, unsupported by the API, stale reporting or verification still pending. Preserve each reason. Displaying all of them as zero invites confusion between 'service time not measured' and 'service time is zero', leading to incorrect scheduling or bans.

12What zero traffic, bad data and bans each mean

Connection management must reduce unproductive costs without unfairly penalizing sources. Closing a temporarily unnecessary connection may simply release resources; concluding that a source sends corrupt data requires more specific evidence. The two situations can warrant different treatment.

The same 0 B/s can have completely different causes

Evaluating a nonresponse requires at least a specific outstanding request, protocol-appropriate service conditions and a healthy local receiving environment. An interval lost to interrupted reporting must not silently be filled in as zero speed.

Even after excluding local interference, a timeout mainly proves that this service attempt did not succeed. Remote load, the network path and protocol scheduling may all contribute. That can justify switching sources or temporary avoidance, but does not inherently establish malicious intent.

Keepalives and late data have different meanings

A keepalive indicates some connection activity but adds no file progress; it must not indefinitely reset expectations for a data request. Conversely, cancelled requests may still produce late data, particularly because Fast Extension explicitly addresses cancellations crossing with responses.^[01][07]

The engine must therefore retain request context: previously requested, cancelled, being answered and never requested are different cases. Connection analysis should use this native information rather than treating all late data as suspect merely because no request is outstanding now.

A hash failure proves a mismatch before it assigns responsibility

Suppose four blocks of a piece came from A, B, C and D, and the final piece hash fails. That shows a problem in the assembled data, not that all four sources intentionally sent bad data. Local writing, memory or state association may also be at fault.

One approach is to isolate suspect data and obtain samples with clearer attribution. libtorrent's parole mechanism requests a whole piece from a peer involved in a failed piece, providing stronger evidence for assessment. This illustrates the practical value of rechecking and narrowing responsibility.^[09]

Let the evidence determine the scope of action

Retransmission of the same error report must not count as another violation. Nor should a source's failure on one content item expand automatically into a whole-subnet ban. When lifting a restriction, confirm that the engine actually removed it, rather than merely clearing its label in the interface.

13How source lists are updated—and how far they can be trusted

Tracker or eD2k server lists provide candidate bootstrap entry points so a downloader need not start entirely from scratch. Longer lists do not necessarily mean more independent sources or better download capability. Lists may overlap, and domain aliases or shared infrastructure can inflate counts.

A source catalog is better understood as a continually revised collection of leads. Record its provenance, acquisition time, which queries actually yielded new sources, and whether those sources later provided service. Being a candidate, previously reachable, a provider of useful sources, and a sender of correct data are distinct levels of accumulated evidence.

Downloading a list only obtains candidates

After an external feed changes, parse candidates and check formats and differences before gradually admitting them into discovery. Useful older catalog versions can remain available; a failed update need not empty the entire source collection. User-pinned or avoided sources also have reasons independent of the feed.

Config publishes one consistent effective view so Components do not make decisions from different list versions. A remote feed may supply endpoints, but must not gain authority to change local policies or execute commands.

An endpoint is input that can trigger a connection

A source record may ultimately cause the downloader to initiate a network connection. If a public source returns a local administration interface, private address or special service port, the client could be used to reach unintended targets. This is the source-discovery equivalent of an SSRF-type problem.

For public Internet sources, address policy must check the actual resolved destination: an ordinary-looking URL may resolve to another address. The same destination restrictions must apply to redirects, retries and fallback resolution. OWASP specifically warns that checking DNS once, then performing an unchecked resolution when connecting, leaves room for bypass.^[17]

If LAN sharing is intended, private-network sources can belong to a separate explicit scope rather than being labeled malicious by a blanket global rule. Public, private and overlay networks require treatment consistent with their own entry points, identities and source-propagation semantics.

Measure source value by new opportunities and actual service

A tracker returning many mostly duplicate or unreachable peers may be less valuable than one returning a few new sources. Observe unique sources per appropriately timed query, reachability ratios and how many sources later provide required data.

The denominators must be explicit too. Never queried means unknown; failures during a local network outage should not lower every source's rating at once. A UDP tracker connect response alone does not mean an announce successfully returned peers: these are separate protocol steps.^[08]

Normal transfer proves only the observed interaction

HTTPS protects the corresponding connection's transport and authenticates the server; a content hash checks data against an expected value. Neither tells us whether the remote operator logs interactions. BitTorrent tracker and peer protocols inherently exchange content identifiers and contact information.^[01]

Thus, even an endpoint that follows the protocol perfectly and sends correct data cannot be ruled out as monitoring or logging on those observations alone. This is a limitation inferred from what is observable: service quality, data integrity and true intent are different questions. Country, ASN, peer ID and high-speed transfer cannot substitute for that distinction.

A client can address visible anomalies such as invalid message lengths, excessive source floods, attributable corrupt data, and dangerous filenames or paths. Source names and metadata should also be rendered as untrusted text. For intentions a peer has not revealed, keeping the answer unknown is more honest than inventing a safety percentage.

14Restoring consistency after pauses, restarts and lost commands

A long-running downloader cannot assume that its management process, engine, database and disk always succeed together. After a pause action, the manager may have recorded the intent before the engine receives it; alternatively, the engine may already have stopped but its reply was lost.

The system must therefore distinguish what it is intended to do from what it is confirmed to have done. This distinction gives pausing tasks, closing connections and post-restart state a coherent recovery path.

Persist intent first, then track whether it takes effect

The manager can persist an operation's intent together with pending work before sending it to the engine. When a reply arrives, update the actual result. If the process stops in between, a restart reveals which operations remain unconfirmed instead of relying on memory-based guesses.

Retries must also identify the same operation. Otherwise, retrying after a network or process failure may duplicate tasks, repeatedly penalize the same evidence, or apply an old command to a new connection. Operation identity, engine startup generation and target state help distinguish these cases.

SQLite stores recoverable facts; Metrics stores changes over time

Tasks, content identities, effective policies, decision-relevant evidence and file-operation records belong in Store because they must remain meaningful after restart. Per-second peer speeds, latency distributions and load curves belong in Metrics, whose storage can follow detailed-retention and aggregation requirements.

SQLite WAL mode allows readers and a writer to proceed concurrently, but still permits only one writer at a time; long-running readers can also impede checkpoint progress. This architecture chooses to coordinate important state writes centrally and keep transactions short. That clarifies data ownership, but does not automatically solve consistency with files and engines outside the database.^[15]

Historical-monitoring failures need not disrupt healthy transfers

Component can report the live summaries needed by Scheduler and Connection Manager directly. Downloads can continue through this path while Metrics temporarily cannot store history. Use bounded buffers and explicit gaps for observation samples so a monitoring failure cannot exhaust memory.

A Store failure to persist important new state has different consequences: new dispatches and persistent policy changes become unreliable. The system should constrain those operations until recording works again. The engine still needs independent real-time protections against malformed messages, failed connections and local resource problems.

Measure durations with a monotonic clock unaffected by wall-clock adjustments; record dates needed across restarts separately. Unobserved service time during restart must not be filled in as source timeouts, and old snapshots must be reconciled against the new engine generation.

15From scattered pieces to complete, recoverable files

Tasks may enter and leave the pool, but acquired data should not disappear with them. Moving a task into observation usually requires only execution changes and saved resume state—not moving its files, much less deleting them.

Incremental allocation and a file's two sizes

Registering many targets first and allocating data only when transfer is needed avoids reserving full-file capacity for every target upfront. The trade-off is that insufficient space may appear mid-download, while allocation and I/O behavior depend on the filesystem and engine.

Distinguish logical size from allocated space. A sparse file can have a large logical length without allocating an equivalent number of physical blocks for unwritten holes. Available disk space also needs to cover more than payload: resume data, the database, temporary files, verification and additional space during final moves.

Incremental writing is flexible; preallocation can reveal capacity problems earlier and reduce allocation fragmentation on some filesystems. This trade-off can be chosen according to the storage device, file scale and usage scenario; one allocation strategy need not be imposed on every task.

Filenames are for display; storage identity must be stable

Working directories can be bucketed by stable content identifiers to avoid overcrowded directories and same-name overwrites. Identifiers must retain protocol semantics; strings from different hash namespaces must not be treated automatically as the same object.

Before full metadata arrives, a magnet task may use a temporary identity. Once content correspondence is established, record changes to task identity and file location so written data remains discoverable after restart. Sanitize original titles and paths as well: source-supplied ../ or absolute paths must not determine local write destinations. The BitTorrent v2 specification explicitly addresses this path handling.^[03]

Transfer completion still requires persistence and final publication

After all required data passes verification, confirm pending writes, native resume state and actual file locations. Showing 100% while data and recovery records are inconsistent may require extensive rechecking after restart, or leave it unclear which data is usable.

On a local filesystem supporting the relevant semantics, rename within one mount can provide an atomic name switch. A move across mount points cannot be completed with the same rename operation alone; it usually requires copying to a temporary destination file, then confirming and publishing it.^[20]

An atomic name switch does not guarantee survival after power loss. On Linux, for example, fsync() synchronizes file data and related metadata, while persisting directory entries may also require syncing the directory. A complete move workflow therefore handles content, location records and the timing of old-copy deletion separately.^[21]

If a cross-device copy fails midway, retaining the old copy and an unfinished-operation record is more reliable than hastily deleting the original. Files still being shared require path coordination with the engine; externally moving a file the engine still uses may break subsequent uploading and resuming.

Consider retention policies only after prolonged source absence

A task with no sources today may gain an online peer tomorrow; content with rare final pieces may recover when someone shares it again. Partial-file cleanup should consider long-term observations, user retention intent, remaining queue opportunities, and any active writer or unconfirmed operation.

If automatic cleanup is needed, first moving candidates into a recoverable quarantine stage, then deciding deletion according to retention time, reduces the cost of mistakes. Slow downloads, a temporarily offline tracker or avoidance of one source do not individually prove that the entire data set has lost its value.

16Trace symptoms back to causes

The value of component boundaries ultimately appears in whether problems can be explained. A 'downloading' label and one speed leave a stalled task open to guesswork. Following content, sources, requests, verification and local load narrows the cause.

The interface should show reasons, not just state names

The download list can show the current phase and main reason for waiting. Task details can add missing ranges, source distribution, request state and local pressure. Connection details should retain what data it provided, when it was observed, why a limit exists and whether the engine has confirmed the operation.

Users can then distinguish waiting for sources or remote queues from local backpressure caused by an HDD that cannot keep up. When source information is unknown, they can also distinguish not yet observed from a field the engine never exposes.

Follow one download through the whole architecture

After the user adds content, Manager saves intent and Search obtains metadata and sources through the corresponding Component. Ready tasks enter limited trials; Engine requests pieces within permitted sources and budgets. Actual progress and local pressure return to Scheduler, while connection service observations return to Connection Manager.

Useful data keeps accumulating while unproductive investment is reduced. New sources can still be discovered, and long-waiting tasks or tasks with rare pieces retain opportunities. Store makes important state recoverable; Metrics enables analysis of past changes. Finally, the selected content passes verification, data and storage state become consistent, and the system publishes the completed files and updates their locations. Pieces verified during the download may also continue to be shared according to the native protocol and user intent.

Clear connections between these responsibilities let a downloader keep making evidence-based choices through changing sources, network waits, disk bottlenecks and process interruptions. The next time speed drops, the system should be able to explain where progress is blocked and what change would let it continue.

References

The following sources were used to check protocols, engine behavior and storage mechanisms; access date: 2026-10-25. Resource calculations and piece matrices illustrate principles; the dynamic download pool and component responsibilities are the architectural design presented in this article.

1. S01
   BitTorrent base protocol | BEP 3

   Content identity, piece hashes, trackers, request pipelining and endgame.

2. S02
   Metadata exchange and magnet links | BEP 9

   Exchange of the info dictionary and location information in magnet links.

3. S03
   BitTorrent v2｜BEP 52

   SHA-256 infohashes, per-file Merkle trees, piece alignment and path handling.

4. S04
   BitTorrent DHT｜BEP 5

   Peer lookup by infohash, routing tables and node information.

5. S05
   Peer Exchange｜BEP 11

   Exchange of other sources through existing peer connections and update limits.

6. S06
   Private torrents | BEP 27

   The private flag and limits on discovery and connection scope.

7. S07
   Fast Extension｜BEP 6

   Allowed-fast, explicit rejection, and the semantics of choking, cancellation and responses.

8. S08
   UDP Tracker｜BEP 15

   Distinct roles of connect, announce and scrape.

9. S09
   libtorrent native observations | Core API

   Peer states, half-open connections, handshakes, pending writes and parole.

10. S10
    libtorrent piece selection

    Official feature documentation. This article cites principles such as rarity and priority only; this older document is not used to establish current internal implementation details.

11. S11
    aMule: the Kad network

    Distributed source discovery, bootstrapping and its relationship to eD2k file transfer.

12. S12
    aMule：eD2k／Kademlia FAQ

    Historical explanations from the official wiki, used here only for mechanisms such as complementary sources, queuing and UDP interactions.

13. S13
    aMule: AICH and corruption recovery

    Locating damaged sub-blocks and preserving correct data when a trusted root and recovery data are available.

14. S14
    aMule: High ID and Low ID

    Inbound reachability and callbacks; this article does not treat IDs as reputation or speed rankings.

15. S15
    SQLite：Write-Ahead Logging

    Concurrent reading and writing, the single writer, checkpoints and the effects of long readers.

16. S16
    Linux kernel: Pressure Stall Information

    Measuring pressure caused by waits for CPU, memory and I/O resources.

17. S17
    OWASP: SSRF prevention

    Destination validation, redirects, DNS rebinding and address consistency at connection time.

18. S18
    qBittorrent: the actual scope of a ban operation

    TransferController at pinned commit d402f87: a source-code example in which banPeersAction calls banIP.

19. S19
    aMule: the role of eD2k servers

    File indexes, source lists and assistance with connection establishment.

20. S20
    Linux man-pages：rename(2)

    Atomic name replacement and the EXDEV limitation across mount points.

21. S21
    Linux man-pages：fsync(2)

    The distinction between file synchronization, directory entries and durability.