A payment flow can appear straightforward: create an order, initialize a transaction, receive confirmation from the payment provider, and mark the order as paid. In a distributed environment, however, each of these operations can fail independently. Webhooks may be delivered more than once, message brokers may become temporarily unavailable, and separate services cannot depend on a single shared database transaction.
I built PayFlow around these failure scenarios, focusing not only on integrating Paystack, but on designing a payment architecture capable of handling duplicate delivery, partial failures, asynchronous communication, and eventual consistency.
The Problem
Payment processing involves several independent systems that need to agree on the state of a transaction.
An order may belong to one service while its payment belongs to another. Paystack processes the transaction externally and later sends a webhook confirming the result. The application then needs to update its own payment state and reliably communicate that change to the order system.
The central problem I explored was:
How can a distributed payment system reliably move an order from pending payment to paid without using shared database transactions or assuming that external events will only be delivered once?
This led me to implement concepts including database-per-service architecture, transactional outbox, idempotent processing, asynchronous messaging, dead-letter queues, server-side payment verification, and correlation-based observability.
System Architecture
Client
|
HTTP / JSON
|
v
API Gateway
Go + Gin
/ \
gRPC gRPC
| |
v v
Order Service Payment Service
| |
| | HTTPS
| v
| Paystack
|
| PostgreSQL
| |
| Transactional Outbox
| |
| v
| RabbitMQ
| |
+<---------------+
Payment Events
PostgreSQL
Order Database
The system consists of three primary services: an API Gateway, Order Service, and Payment Service.
The API Gateway handles external HTTP communication while the internal services communicate synchronously through gRPC. Payment state changes are propagated asynchronously through RabbitMQ, and each domain owns its own PostgreSQL database.
API Gateway
The API Gateway is the external entry point into PayFlow.
Built with Gin, it accepts HTTP/JSON requests and translates them into internal gRPC calls. It deliberately contains no payment or order business logic and has no direct database access.
The gateway also receives Paystack webhooks. Instead of parsing the webhook before forwarding it, the gateway preserves the raw request body and Paystack signature and sends both to the Payment Service through gRPC.
This allows the Payment Service to perform cryptographic verification against the exact payload originally signed by Paystack.
Order Service
The Order Service owns order data and the order lifecycle.
When an order is created, the service validates the supplied information, generates the order identifier, and stores the order in its PostgreSQL database with a pending payment state.
The Order Service also acts as a RabbitMQ consumer.
When it receives a valid payment.succeeded event, it updates the corresponding order from:
PENDING_PAYMENT
|
v
PAID
Payment events are processed idempotently so duplicate RabbitMQ deliveries cannot repeatedly apply the same state transition.
Payment Service
The Payment Service owns payment state and all communication with Paystack.
When a payment is initiated, the service validates the order information, customer email, amount, and currency before creating a PENDING payment record.
It then initializes the transaction with Paystack and uses the payment’s own UUID as the provider transaction reference.
Following successful initialization, the payment moves to:
PENDING | v INITIALIZED
The Paystack authorization URL is returned to the caller so the customer can complete the transaction.
If provider initialization fails, the payment attempt remains recorded and is marked FAILED, preserving an audit trail rather than silently removing the attempt.
Secure Payment Verification
PayFlow does not treat an incoming webhook as sufficient evidence that a payment succeeded.
When Paystack sends a webhook, the Payment Service first validates its authenticity using HMAC-SHA512 signature verification.
The service then independently contacts Paystack’s transaction verification endpoint using the transaction reference.
The verified amount and currency are compared with the values originally recorded by PayFlow before the payment is accepted as successful.
The flow therefore becomes:
Paystack Webhook
|
v
HMAC-SHA512 Verification
|
v
Extract Transaction Reference
|
v
Server-side Paystack Verification
|
v
Validate Amount + Currency
|
v
Update Internal Payment State
This keeps payment confirmation on the server side rather than trusting client-side information or the webhook payload alone.
Transactional Outbox
A major reliability problem in event-driven systems is the dual-write problem.
Consider the following sequence:
1. Update payment in PostgreSQL 2. Publish payment.succeeded to RabbitMQ
If step one succeeds but RabbitMQ becomes unavailable before step two completes, the Payment Service records the payment as successful while the Order Service never learns about it.
PayFlow addresses this using the Transactional Outbox Pattern.
Instead of publishing directly to RabbitMQ during webhook processing, the Payment Service performs:
PostgreSQL Transaction
|
+--------+--------+
| |
v v
Payment = SUCCESS Create Outbox Event
Both operations occur within the same database transaction.
Either both succeed or neither does.
A background outbox worker then periodically retrieves unpublished events from the database and publishes them to RabbitMQ.
Verified Payment
|
v
PostgreSQL Transaction
|
+--> Payment = SUCCESS
|
+--> Outbox Event
|
v
Outbox Worker
|
v
RabbitMQ
|
v
Order Service
|
v
Order = PAID
This removes RabbitMQ availability from the critical database transaction path and ensures events remain durable when temporary broker failures occur.
Concurrency-Safe Event Processing
The outbox worker polls for unpublished events in batches and uses PostgreSQL’s:
FOR UPDATE SKIP LOCKED
when claiming rows.
This allows multiple workers to operate concurrently while preventing them from claiming the same event simultaneously.
Failed publication attempts remain available for later processing rather than being discarded.
Idempotency
PayFlow assumes at-least-once delivery, not exactly-once delivery.
This is important because both payment webhooks and RabbitMQ messages can be delivered more than once.
For webhook processing, the Payment Service maintains processed webhook records. The processed marker and payment update occur atomically within the same database transaction.
For RabbitMQ events, each domain event contains a unique event identifier. The Order Service stores processed event IDs so that a redelivered message cannot apply the same operation multiple times.
The architecture therefore treats duplicate delivery as an expected condition rather than an exceptional one.
Event-Driven Communication
PayFlow uses a durable RabbitMQ topic exchange:
payflow.events
The Order Service subscribes to payment-related events using:
payment.*
The primary domain events include:
payment.succeeded payment.failed
Messages use manual acknowledgements.
The consumer distinguishes between transient and permanent failures:
Transient Failure
|
v
Retry
Permanent Failure
|
v
Dead Letter Queue
Invalid or permanently unprocessable events are routed to a Dead Letter Queue (DLQ) instead of being silently discarded.
Database-per-Service Architecture
PayFlow follows the database-per-service principle.
The Order and Payment services have separate PostgreSQL databases:
payflow_order payflow_payment
The Order Service owns:
orders processed_events
The Payment Service owns:
payments processed_webhook_events outbox_events
Although the Payment Service stores an order_id, there is intentionally no cross-database foreign key connecting it to the Order Service.
The relationship between the two domains is maintained through service communication and events rather than shared database ownership.
Database schema changes are managed using explicit golang-migrate migrations instead of relying on ORM auto-migration.
Observability
PayFlow implements correlation IDs to make transactions traceable across service boundaries.
A correlation ID originates at the API Gateway and is propagated through:
HTTP Request
|
v
gRPC Metadata
|
v
Service Logs
|
v
RabbitMQ Event
|
v
Consumer Logs
This makes it possible to follow one logical operation even when it crosses several services and transitions from synchronous to asynchronous processing.
The system also exposes:
GET /health GET /ready
The health endpoint provides process-level liveness information, while readiness checks verify whether required dependencies are actually available.
The internal services periodically check their PostgreSQL connections and report themselves as unavailable when their database dependency cannot be reached.
Payment Lifecycle
The main payment lifecycle is:
PENDING | | Paystack initialization v INITIALIZED | | Webhook received | Signature verified | Transaction independently verified | Amount + currency reconciled v SUCCESS | | payment.succeeded v RabbitMQ | v Order Service | v ORDER PAID
Failed payment initialization or verified unsuccessful transactions can move the payment to FAILED.
Testing
The project includes tests across the major layers of the system, covering areas such as payment and order creation, gRPC handlers, webhook processing, Paystack client behaviour, repository operations, RabbitMQ messaging, API Gateway handlers, correlation propagation, health checks, and transactional outbox processing.
The repository also includes concurrency-focused tests around duplicate processing and competing outbox workers, reflecting the distributed-system failure scenarios the architecture is designed to handle.
Local Development & Deployment
The complete local environment is containerized using Docker Compose.
The environment consists of:
API Gateway Order Service Payment Service PostgreSQL RabbitMQ
RabbitMQ’s management interface can also be used to inspect exchanges, queues, message delivery, and dead-letter behaviour.
The project additionally includes a development-only fake Paystack server, allowing the payment flow to be exercised locally without depending on real Paystack credentials.
What I Learned
PayFlow strengthened my understanding that the difficult part of payment processing is not simply integrating a payment API. The harder challenge is maintaining trustworthy state when independent systems communicate across unreliable boundaries.
Building the project required me to think through questions such as what happens when the database succeeds but message publication fails, how duplicate webhooks should be handled, how multiple workers can safely process events concurrently, how data ownership should work across services, and how one transaction can be traced across synchronous and asynchronous communication.
Working through these problems gave me practical experience with distributed systems reliability, eventual consistency, concurrency, fault handling, event-driven architecture, service boundaries, and production-oriented backend design.
Known Limitation & Future Improvements
The current RabbitMQ client connection does not automatically recover after an established broker connection is lost. If RabbitMQ goes offline and later becomes available again, the relevant services currently need to restart to establish fresh connections.
The transactional outbox still protects unpublished events from being lost because they remain persisted in PostgreSQL until publication succeeds.
Future improvements include automatic RabbitMQ reconnection, exponential retry and backoff, metrics and distributed tracing, API authentication and authorization, automated payment reconciliation, and Kubernetes-based deployment.
