Transaction Management with Dependent APIs
Failures are unavoidable in any application, be it distributed or centralized.
Important thing is how our application recover from failure and customer/business is least impacted.
Each distributed application at a high level does one or more following things in any order.
- Invoke Dependent API
- Publish Event
- Update state of resources.
So in distributed applications, each operation is spread across multiple applications. In case of failure, we need to make sure that either all operations within Transaction Boundary are performed or none are performed.
Let's talk about the first use case —
How to manage the transaction with dependent APIs?
Dependent APIs can fail for either transient or non-transient errors. Based on the types of errors we can decide our approach.
Transient errors are network errors like connection, timeout, etc. and there are chances that retry can fix that. While Non-transient are errors are the ones that won’t be successful in retry like validation errors, authorization errors, etc.
Non-transient error, we can directly put it into a dead-letter queue(DLQ) for further analysis, but for a transient error, we can retry. Retry mechanisms can vary based on the nature of dependent APIs.
Let's look at the different types of APIs we can have…
Idempotent APIs
If dependent API is idempotent, it is safe to retry as this doesn’t create any side effects in the application. Definition of idempotent API :
An idempotent API is a method that can be invoked many times without the different outcomes. .
Every idempotent API uses the idempotent key to identify if the operation was performed or not. While retrying we need to make sure to pass the same idempotent key.
For example, when an order application receives an Order Cancellation event from Kafka Topic, it makes a call to payment & stock application. Both APIs are idempotent. If failure happens at any stage be it in payment or stock or while sending acknowledge back to Kafka. In case of failure, the order service won’t commit the offset, so next time the order service will get the same event and since all APIs are idempotent we can call payment & stock without any side effects. That is the advantage of integrating idempotent APIs.
In a distributed application, it is considered a good practice to design every operation as an idempotent one.
Non Idempotent APIs, support reading state —
Let's talk about the scenario, where dependent API is not idempotent. In that case, we can not simply retry the operation. Since we are not sure whether the first attempt was successful or not.
But if there is an API available to read the state of a resource then we can use that to verify the state of a resource. If the resource is already in the target state then we can skip API call.
Lets take the same example, suppose payment refund API is not idempotent anymore. So how we can maintain transactions?
If there is another API that can give us the status of payment, then we first call that method to get the payment status and if it is already refunded then we will skip the refund API call otherwise we will make a call.
When you are dealing with non-idempotent APIs, there will be always an overhead to maintain transactionality at the consumer end. Here in this case we need to make extra get status call every time when the event is received. So this will also add some latency to our operation. That is why we need to design every API with idempotency in mind.
Non-Idempotent APIs- without reading state
This is a very tricky situation and resolution varies based on use cases. But there are no straightforward solutions for such cases.
For example, in the above case, if get payment status API is not available then another way is to keep track of the refund APIs that were called against orderId and every time check against that. But this is not a full-proof solution because it very much depends on when you are storing that information before/after API call etc.
It's very difficult to maintain transactionality in such cases. We should ask to make APIs as idempotent or provide other APIs with can support read operation.
In summary,
- If API is idempotent — nothing like it, retry without worry
- If API is non-idempotent — support reading state than Read-Validate-Call.
- If API is non-idempotent — not support the reading state then the solution depends on the use case.
This is one use case, in the next article, we will see how to manage transactions with Datastore.
Happy learning..!!!
