An event feed and an event index solve different problems. A feed helps an application react to new observations. An index attempts to maintain a coherent, queryable record across restarts, interruptions, and changing source state. Confusing the two can leave gaps that remain invisible until somebody asks for a historical report.
This guide outlines a practical indexing design for a small application. It does not prescribe one universal architecture or a provider-specific implementation. Begin with a single event family, a limited historical range, and a clear statement of what “processed” means. Expand only after your recovery process produces the same result as an uninterrupted run.
Define the record your application needs
Write the schema before choosing a subscription. A useful event record might include network identity, source reference, transaction identifier, event position, decoded fields, and an observation status. Preserve the raw payload or a reproducible reference when your debugging and retention policies justify it.
Separate source facts from application interpretation. A decoded transfer-like event may support one technical observation, while your product attaches an additional label or category. Store those layers separately so a later decoding change does not masquerade as a change in the underlying source record.
Choose the first event family narrowly. It is easier to establish correct replay and reconciliation for a small dataset than to diagnose a broad index whose parsing rules are still changing. The use-case overview offers examples of features that can begin with deliberately limited data coverage.
Understand why a live subscription is not a history store
Geth documents that its subscriptions deliver current events rather than past events and are tied to a connection. It also describes how log notifications can report removed records during a chain reorganization. These are important boundaries for an application that needs complete, consistent history. See the Geth real-time events guide for the exact behavior.
Design a recovery path that does not depend on the live connection having remained uninterrupted. A new connection can help you receive new observations, but your application still needs a strategy for checking what happened during a gap.
Keep subscription state distinct from index completeness. A connected socket says something about transport. It does not prove that every relevant historical record is present in your database. Your operational view should make those different claims separately visible.
Distinguish observed progress from committed progress
Track the newest source reference you have seen separately from the range whose records have been fully processed and persisted. Seeing an event is not the same as completing every downstream operation associated with it. Naming those milestones precisely prevents premature checkpoint advancement.
Advance a durable checkpoint only after the corresponding processing work satisfies your application's guarantees. Where several records belong to one range, define whether the checkpoint refers to the entire range or to individual positions. Make this rule part of the storage design rather than an informal assumption in a worker loop.
Test a crash at each boundary. Interrupt before saving records, after saving some records, and immediately before updating the checkpoint. On restart, the final state should converge to the intended result without silently skipping unfinished work.
Make replay safe before making it fast
Assume your own recovery process may deliver the same observation more than once. Select an identity that distinguishes the records your application needs and write an explicit rule for updating existing records. This is a design requirement even without claiming any particular provider delivery guarantee.
Keep external side effects separate from index updates. Reprocessing an event should not accidentally send the same notification repeatedly or invoke an unrelated action twice. A record of completed side effects may need its own identity and lifecycle rather than relying on the presence of a display row.
Use a replay fixture. Process a known set once, process it again, and compare the resulting database state and side-effect log. Then process overlapping subsets in a different order. These simple tests reveal whether the implementation truly treats repeat processing as an ordinary operation.
Give reorganizations a deliberate representation
A historical record should not become irrevocably canonical just because your application observed it once. Decide how you represent a record whose source context changes. Your model may retain the observation while updating its canonical status, or it may rebuild an affected range from source evidence.
Avoid destroying the only evidence needed to explain a correction. For some applications, an audit trail of the original observation and its later status is useful. For others, a reconciled current view is enough. Choose based on the product's actual needs and retention policy, not on a generic desire to store everything.
Keep correction handling separate from ordinary duplication. Receiving the same logical event again and learning that a previous observation no longer belongs to the selected history are different situations. Tests should cover both so one generic “already exists” branch does not suppress necessary updates.
Reconcile bounded ranges with clear completion rules
Choose a manageable range size for historical retrieval and record which range is being processed. Do not infer completeness merely because one response contains some records. Your implementation must follow the source's documented pagination or range behavior and account for any limits that apply to the query.
When a range fails, preserve its incomplete status. Retry according to a bounded policy and leave enough information for an operator to inspect persistent failures. Advancing beyond a failed range can be acceptable only when the index explicitly represents the gap and downstream consumers understand it.
The retry guide discusses budgets and error classification. Apply those ideas to backfills as well as interactive requests. A historical job can create substantial load if every failed chunk independently retries without a shared concurrency limit.
Measure completeness instead of celebrating activity
A busy stream of log messages does not demonstrate a healthy index. Track the last fully processed reference, unresolved gaps, failed ranges, and the age of outstanding work. These measures describe whether the dataset can answer the questions your application promises to support.
Include a reconciliation check in your normal operating routine. Compare a selected range against the source and inspect differences. The purpose is not to produce an impressive dashboard statistic but to find assumptions that quietly diverged from the data.
Set alerts around actionable states. An alert should tell the responsible person which range or processing stage needs attention and what evidence is available. A vague “index unhealthy” message can create urgency without making recovery easier.
Document how consumers should interpret the index
A downstream screen or report should know the index's coverage and update status. Expose those fields as part of the query response or accompanying metadata. A report built from an incomplete range should not appear indistinguishable from one built from a reconciled dataset.
The consumer contract
Write a short operational contract: which networks and event families are included, where history begins, how corrections are represented, and how fresh the completed data is. This document helps both engineers and product reviewers understand the limits of the feature.
Conclusion: recovery is part of the index
A dependable event index separates live observation, durable processing, canonical interpretation, and downstream side effects. It can replay safely, reconcile gaps, and explain its coverage to consumers. Those capabilities matter more than receiving a large volume of messages during a happy-path demonstration.
Start with a bounded dataset and prove that interrupted processing converges to the same result as an uninterrupted run. Then scale the ranges and event families carefully. The transport comparison helps place subscriptions within that larger architecture rather than treating a persistent connection as the entire solution.