Multi-Collection Updates and ACID Compliance Using Context and Transactions in MongoDB Using Go

Published in

Better Programming

13 min readOct 27, 2023

In the world of web development and database management, working with multiple database collections is something that everyone comes around to once in a while. Recently, during a conversation with a friend, this subject triggered a fascinating discussion. We were discussing whether to divide the API endpoint for a feature into two independent HTTP calls (POST and PATCH) or create a single one to perform operations on multiple collections together. We wanted to avoid the infamous “N+1” Problem. Let me give you a brief introduction to that.

The “N+1” problem in databases refers to a situation where an initial query (the “1”) fetches a set of data, but subsequent related queries (the “+1”) are needed to retrieve additional, related information. This can lead to a large number of queries being executed, which can be inefficient and slow down the application.

For example, imagine you have a list of users, and for each user, you want to retrieve their associated orders. If you retrieve the list of users first and then make a separate query for each user’s orders, you’ll end up with N+1 queries, where N is the number of users. This can be a performance bottleneck, especially when dealing with large datasets.

To address the N+1 problem, techniques like eager loading (fetching related data in a single query) or using database joins (combining related data into a single result set) can be employed. These approaches help reduce the number of queries needed and improve the overall performance of the application.

OK, where were we? Since the two operations were closely related, we decided to use the latter one to perform a single API call (PATCH) and combine the related operations on multiple collections into a single result set.

This is where we thought, what if one operation passed and another failed while executing? What happens then? How would the database react? What would be the result?

Indeed, these were good questions. Hence, I decided to write a blog about it.

TL;DR

The short answer to the above questions is to use Contexts and Transactions to make operation ACID compliant. And yes, MongoDB offers ACID compliance using Transactions. Read more here

Longer Answer

Before we delve into this topic, we need to know a few concepts. Let’s go through each of them

Understanding Contexts

Contexts play a pivotal role in maintaining the flow of information and ensuring that operations are executed in an organized manner. In the context of our discussion, they provide a structured environment for managing sessions and transactions. Properly handling contexts can significantly enhance the robustness and reliability of our applications.

ACID Compliant Databases

To address the challenges posed by multi-collection updates, it’s imperative to rely on ACID-compliant databases. ACID (Atomicity, Consistency, Isolation, Durability) is a set of properties that guarantee reliable and secure transaction processing. These databases ensure that transactions are executed reliably, even in the event of failures or errors.

Transactions: Ensuring Data Integrity

Transactions are the backbone of maintaining data integrity in a database system. They enable a series of operations to be executed as a single atomic unit. In case of any failure within the transaction, the entire operation can be rolled back to its initial state, preventing any partial updates or inconsistencies.

Enough Theory, I Need To See Some Code

Let’s cement the knowledge by implementing it via a small project. We will create a project where we will provide an API for managing users and their orders. It allows users to create accounts with an initial balance and make orders, which deduct the specified amount from their balance while recording the transaction details.

We’ll also leverage the Gin framework to handle HTTP requests and responses. The application will manage users, orders, and the placement of orders.

Prerequisites

Before we start, make sure you have the following installed:

Go
Create an account on Mongo Atlas
The Go driver for MongoDB (go.mongodb.org/mongo-driver)
The Gin framework (github.com/gin-gonic/gin)

Project Structure

Create a Go project with the following structure:

mongo-transactions/
|-- main.go
|-- go.mod

Setting Up MongoDB Connection

Let’s start by setting up the MongoDB connection in the main.go file:

main.go

package mainimport (
	"context"
	"fmt"
	"log"
	"net/http"
	"time"	"github.com/gin-gonic/gin"
	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/bson/primitive"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"
	"go.mongodb.org/mongo-driver/mongo/readconcern"
	"go.mongodb.org/mongo-driver/mongo/writeconcern"
)var client *mongo.Client
var database *mongo.Database// ... (other code will be added here)

Initializing MongoDB Client

Next, we’ll add an initialization function to connect to our MongoDB instance and set up the database:

func init() {
   // Set up the MongoDB client
   clientOptions := options.Client().ApplyURI(os.Getenv("MONGO_URL"))
   var err error
   client, err = mongo.Connect(context.TODO(), clientOptions)
   if err != nil {
    log.Fatal(err)
   }
  
   // Check the connection
   err = client.Ping(context.TODO(), nil)
   if err != nil {
    log.Fatal(err)
   }
  
   database = client.Database("my_database")
   fmt.Println("connected to database")
}

Replace my_database with the name of your MongoDB database.

Defining Data Models

We’ll define three data models: User ,Order , and PlaceOrderRequest which is the schema for HTTP POST requests for creating orders.

type User struct {
	ID      primitive.ObjectID `bson:"_id,omitempty"`
	Name    string             `bson:"name,omitempty"`
	Balance int                `bson:"balance,omitempty"`
}

type Order struct {
       ID       primitive.ObjectID `bson:"_id,omitempty"`
       UserID   primitive.ObjectID `bson:"user_id,omitempty"`
       Amount   int                `bson:"amount,omitempty"`
       DateTime time.Time          `bson:"datetime,omitempty"`
}

type PlaceOrderRequest struct {
       UserID primitive.ObjectID `bson:"user_id,omitempty" json:"user_id"`
       Amount int                `bson:"amount,omitempty"`
}

Handling Place Order Request

Now, let’s handle the request for placing an order using a transaction:

func placeOrderHandler(c *gin.Context) {
   var orderRequest PlaceOrderRequest
   if err := c.ShouldBindJSON(&orderRequest); err != nil {
    c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
    return
   }
  
   // Step 1: Update user balance
   userID := orderRequest.UserID
   orderAmount := orderRequest.Amount
  
   userCollection := database.Collection("users")
   userFilter := bson.M{"_id": userID}
   update := bson.M{"$inc": bson.M{"balance": -orderAmount}}
   _, err := userCollection.UpdateOne(context.Background(), userFilter, update)
   if err != nil {
    c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to update user balance: %s", err)})
    return
   }
  
   // Step 2: Create order
   order := Order{
    UserID:   userID,
    Amount:   orderAmount,
    DateTime: time.Now(),
   }
  
   orderCollection := database.Collection("orders")
   _, err = orderCollection.InsertOne(context.Background(), order)
  
   if err != nil {
    c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create order"})
    return
   }
  
   c.JSON(http.StatusOK, gin.H{"message": "Order placed successfully"})
}

The placeOrderHandler function will be responsible for processing the request to place an order with a user-defined amount.

Handling Order and User Retrieval

Next, let’s implement handlers for retrieving orders and users:

func getAllOrdersHandler(c *gin.Context) {
     orders := []Order{}
     orderCollection := database.Collection("orders")
     cursor, err := orderCollection.Find(context.TODO(), bson.M{})
     if err != nil {
      c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to fetch orders"})
      return
     }
     defer cursor.Close(context.TODO())
    
     for cursor.Next(context.TODO()) {
      var order Order
      if err := cursor.Decode(&order); err != nil {
       c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to decode order"})
       return
      }
      orders = append(orders, order)
     }
    
     c.JSON(http.StatusOK, gin.H{"orders": orders})
}

func getAllUsersHandler(c *gin.Context) {
     users := []User{}
    
     userCollection := database.Collection("users")
     cursor, err := userCollection.Find(context.TODO(), bson.M{})
     if err != nil {
      c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to fetch users"})
      return
     }
     defer cursor.Close(context.TODO())
    
     for cursor.Next(context.TODO()) {
      var user User
      if err := cursor.Decode(&user); err != nil {
       c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to decode user"})
       return
      }
      users = append(users, user)
     }
    
     c.JSON(http.StatusOK, gin.H{"users": users})
}

These functions will retrieve all orders and users from the respective collections.

Creating a New User

We’ll also create a handler for adding a new user:

func createUserHandler(c *gin.Context) {
     // Parse JSON request body
     var newUser User
     if err := c.ShouldBindJSON(&newUser); err != nil {
      c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
      return
     }
    
     // Insert the new user into the database
     userCollection := database.Collection("users")
     _, err := userCollection.InsertOne(context.TODO(), newUser)
     if err != nil {
      c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create user"})
      return
     }
    
     c.JSON(http.StatusOK, gin.H{"message": "User created successfully"})
}

This function will parse the JSON request body and insert the new user into the database.

Running the Server

Finally, let’s set up the main function to run the server:

func main() {
	router := gin.New()
	router.POST("/users", createUserHandler)
        router.GET("/users", getAllUsersHandler)
        router.GET("/orders", getAllOrdersHandler)
        router.POST("/place_order", placeOrderHandler)
        router.Run(":9090")
        }

This will start the HTTP server on port 9090 and register the defined routes.

Upon confirming the readiness of our server, let’s proceed with testing the functionality.

Firstly, we’ll create a user by making a POST request to the endpoint http://localhost:9090/users with the following data in the request body:

{
  "name": "John Doe",
  "balance": 1000
}

The expected response will be:

{
 "message": "User created successfully"
}

Next, we’ll retrieve the user information:

Request:

GET /users

Response:

{
 "users": [
  {
   "ID": "6537cb4f8af49a04839d1ac6",
   "Name": "John Doe",
   "Balance": 1000
  }
 ]
}

With the user successfully created, we’ll proceed to place an order for John, using his unique ID 6537cb4f8af49a04839d1ac6.

To place the order, we’ll perform a POST request to http://localhost:9090/place_order with the following request body:

{
 "user_id": "6537cb4f8af49a04839d1ac6",
 "amount": 100
}

Upon a successful order placement, the response will be:

{
 "message": "Order placed successfully"
}

Subsequently, we’ll check the orders to ensure the transaction was successful:

Request:

GET /orders

Response:

{
 "orders": [
  {
   "ID": "6537cb568af49a04839d1ac7",
   "UserID": "6537cb4f8af49a04839d1ac6",
   "Amount": 100,
   "DateTime": "2023-10-24T13:49:10.749Z"
  }
 ]
}

To verify if the user’s deposit amount has been appropriately deducted, we’ll retrieve the user details once more:

Request:

GET /users

Response:

{
 "users": [
  {
   "ID": "6537cb4f8af49a04839d1ac6",
   "Name": "John Doe",
   "Balance": 900
  }
 ]
}

This confirms that the user’s balance was successfully updated to reflect a deduction of 100, as intended.

However, let’s explore a hypothetical scenario where, after deducting the amount from the user’s account, an issue arises during the order creation process.

We’ll deliberately induce an error during the order creation to observe the outcome. First, we’ll update the code to trigger this scenario.

func placeOrderHandler(c *gin.Context) {
 var orderRequest PlaceOrderRequest
 if err := c.ShouldBindJSON(&orderRequest); err != nil {
  c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
  return
 }

 // Update user balance
 userID := orderRequest.UserID
 orderAmount := orderRequest.Amount

 userCollection := database.Collection("users")
 userFilter := bson.M{"_id": userID}
 update := bson.M{"$inc": bson.M{"balance": -orderAmount}}
 _, err := userCollection.UpdateOne(context.Background(), userFilter, update)
 if err != nil {
  c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to update user balance: %s", err)})
  return
 }

 // Create order
-  order := Order{
-  UserID:   userID,
-   Amount:   orderAmount,
-  DateTime: time.Now(),
-  }

-  orderCollection := database.Collection("orders")
-  _, err = orderCollection.InsertOne(context.Background(), order)

- if err != nil {
+ // Failing deliberately
+ if true { 
  c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create order"})
  return
 }

 c.JSON(http.StatusOK, gin.H{"message": "Order placed successfully"})
}

Now that we have tweaked our code to fail let’s proceed to create another order for the same user with the ID 6537cb4f8af49a04839d1ac6.

Request:

POST /place_order

Body:

{
 "user_id": "6537cb4f8af49a04839d1ac6",
 "amount": 300
}

As anticipated, this time an error occurs:

{
 "error": "Failed to create order"
}

Now, let’s inspect the current state of the database, specifically the ‘users’ and ‘orders’ collections. Since the operation failed, we shouldn’t observe any new orders, and no amount should have been deducted from the user’s account.

Let’s begin by checking if the order was created:

Request:

GET /orders

Response:

{
 "orders": [
  {
   "ID": "6537ce95d2989b4d3499eb2a",
   "UserID": "6537cb4f8af49a04839d1ac6",
   "Amount": 100,
   "DateTime": "2023-10-24T14:03:01.104Z"
  }
 ]
}

It appears that the order did not go through, which is expected given the encountered error. These situations are not uncommon in software development.

Now, let’s verify whether the user’s account balance was deducted:

Request:

GET /users

Response:

{
 "users": [
  {
   "ID": "6537cb4f8af49a04839d1ac6",
   "Name": "John Doe",
   "Balance": 600
  }
 ]
}

Oh, it looks like the user account was deducted. That is bad. So why did this happen?

Unfortunately, it seems that the user’s account was indeed deducted. This is an undesirable outcome. The reason for this lies in the fact that we’re performing operations on multiple collections without configuring the operation as atomic or part of a single transaction. Consequently, MongoDB treated them as two separate operations, and when one succeeded, it executed.

In standard scenarios, where we primarily update a single collection at a time, this isn’t problematic. However, there are instances, especially when working with intricate data structures, where it becomes necessary to operate on multiple collections within the scope of a single HTTP operation. This is where the complexity arises.

To address this, we turn to MongoDB transactions. Transactions allow us to perform multiple operations in an atomic manner, ensuring either all operations succeed or none at all. This guarantees data integrity, even in complex scenarios like the one we just encountered.

Inside the transaction, you can perform operations like updating the user’s balance and creating an order. In case of any errors, you can abort the transaction.

Performing Transactions

To do this, we need to update our placeOrderHandler function

func placeOrderHandler(c *gin.Context) {
 var orderRequest PlaceOrderRequest
 if err := c.ShouldBindJSON(&orderRequest); err != nil {
  c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
  return
 }

+ // Start session
+  session, err := client.StartSession()
+  if err != nil {
+  c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to start session"})
+  return
+ }
+ defer session.EndSession(context.Background())

+ // Create a new mongo session context
+ sessionContext := mongo.NewSessionContext(context.Background(), session)


+ // Begin transaction
+ err = session.StartTransaction(
+   options.Transaction().
+   SetReadConcern(readconcern.Snapshot()).
+   SetWriteConcern(writeconcern.Majority()),
+ )
+ if err != nil {
+ c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to start transaction"})
+  return
+ }

 // Update user balance
 userID := orderRequest.UserID
 orderAmount := orderRequest.Amount

 userCollection := database.Collection("users")
 userFilter := bson.M{"_id": userID}
 update := bson.M{"$inc": bson.M{"balance": -orderAmount}}
- _, err := userCollection.UpdateOne(context.Background(), userFilter, update)
+ _, err = userCollection.UpdateOne(sessionContext, userFilter, update)
 if err != nil {
+  session.AbortTransaction(ctx)
-  c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to update user balance: %s", err)})
+  c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to update user balance: %s, rolling back...", err)})
   return
 }

 // Create order
 order := Order{
  UserID:   userID,
  Amount:   orderAmount,
  DateTime: time.Now(),
 }

 orderCollection := database.Collection("orders")
-  _, err = orderCollection.InsertOne(context.Background(), order)
+  _, err = orderCollection.InsertOne(sessionContext, order)
 if err != nil {
+  session.AbortTransaction(ctx)
-  c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to create order: %s", err)})
+  c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to create order, rolling back...", err)})
   return
 }

+// Commit transaction
+ if err = session.CommitTransaction(context.TODO()); err != nil {
+   session.AbortTransaction(ctx
+   handleTransactionError(session, sessionContext, c, "Failed to commit transaction")
+   c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to commit transaction, rolling back..."})
+   return
 }

 c.JSON(http.StatusOK, gin.H{"message": "Order placed successfully"})
}

Here are the few things that we did here

Starting a Session (session, err := client.StartSession()):

A session is started with the MongoDB server. A session is a context in which a set of operations can be performed. It ensures that these operations are executed consistently and in isolation from other operations.

2. Creating a Session Context (sessionContext := mongo.NewSessionContext(context.Background(), session)):

A session context is created using the session. This context allows us to associate the session with database operations. It acts as a context that is aware of the session’s lifecycle.

3. Beginning a Transaction (err = session.StartTransaction(...)):

A transaction is initiated within the session context. This marks the start of a set of operations that should be executed atomically. The transaction is configured with read and write concerns, ensuring consistency and durability.

4. Using Session Context for Database Operations:

All subsequent database operations (userCollection.UpdateOne and orderCollection.InsertOne) are performed using the session context. This means that these operations are now part of the transaction initiated earlier.

5. Error Handling and Transaction Rollback:

If an error occurs during any of the database operations, the transaction is aborted (session.AbortTransaction(ctx)) to ensure that none of the changes made in the transaction are persisted to the database.
Additionally, an error message is returned to the client indicating the failure and that a rollback is taking place.

6. Committing the Transaction (if err = session.CommitTransaction(context.TODO()); err != nil {...}):

If all database operations within the transaction are successful, the transaction is committed. This means that all changes made in the transaction are now finalized and will be persisted in the database.

7. Deferred Session Cleanup (defer session.EndSession(context.Background())):

The session is deferred to ensure it is properly cleaned up after the function execution. This is important to prevent resource leaks.

Now that we’ve made enhancements to our code let’s re-create the scenario to simulate an error. I’ll once again comment on the order creation section, just as we did before, and execute the code.

First, let’s observe the current state of our users and orders:

Users

{
 "users": [
  {
   "ID": "6537cb4f8af49a04839d1ac6",
   "Name": "John Doe",
   "Balance": 600
  }
 ]
}

Orders

{
 "orders": [
  {
   "ID": "6537ce95d2989b4d3499eb2a",
   "UserID": "6537cb4f8af49a04839d1ac6",
   "Amount": 100,
   "DateTime": "2023-10-24T14:03:01.104Z"
  }
 ]
}

Next, I’ll reintroduce a minor issue again in placeOrderHandler function, this time focusing on the order creation process:

... redacted
// Create order
 // order := Order{
 //  UserID:   userID,
 //  Amount:   orderAmount,
 //  DateTime: time.Now(),
 // }

 // orderCollection := database.Collection("orders")

 // _, err = orderCollection.InsertOne(sessionContext, order)

 if true {
  handleTransactionError(session, sessionContext, c, "Failed to create order")
  return
 }
... redacted

Now, let’s observe the outcome when attempting to place a new order:

Request:

POST /place_order

Body:

{
 "user_id": "6537cb4f8af49a04839d1ac6",
 "amount": 300
}

As anticipated, an error has been raised, which is the expected behavior:

{
 "error": "Failed to create order"
}

Now, let’s examine the current state of the orders and users collection.

Upon inspecting the orders, we observe the following:

Request:

GET /orders

Response:

{
 "orders": [
  {
   "ID": "6537ce95d2989b4d3499eb2a",
   "UserID": "6537cb4f8af49a04839d1ac6",
   "Amount": 100,
   "DateTime": "2023-10-24T14:03:01.104Z"
  }
 ]
}

Looks like the order did not go through. OK fine. Now, let’s ascertain if the user’s account balance was affected:

Request:

GET /users

Response:

{
 "users": [
  {
   "ID": "6537cb4f8af49a04839d1ac6",
   "Name": "John Doe",
   "Balance": 600
  }
 ]
}

Woo Hoo.. This is precisely the outcome we aimed for — an “all-or-nothing” transaction. MongoDB transactions, baby!

So, let’s quickly discuss what happened here. We are treating these insets and updates as a single transaction. Hence, unlike before, mongo does not commit changes immediately. It keeps track of the changes and only commits them when you explicitly issue a commit command. If any error occurs during the transaction, mongo will roll back the operations that were part of the transaction.

Now, let’s promptly fix the issue we had introduced previously and reattempt the operation:

Request:

POST /place_order

Body:

{
 "user_id": "6537cb4f8af49a04839d1ac6",
 "amount": 300
}

Success! We receive the confirmation:

{
 "message": "Order placed successfully"
}

Subsequently, let’s examine the state of the orders and users collections:

Request:

GET /orders

Response:

{
 "orders": [
  {
   "ID": "6537ce95d2989b4d3499eb2a",
   "UserID": "6537cb4f8af49a04839d1ac6",
   "Amount": 100,
   "DateTime": "2023-10-24T14:03:01.104Z"
  },
  {
   "ID": "6537d58e215cf6268d15e1b3",
   "UserID": "6537cb4f8af49a04839d1ac6",
   "Amount": 300,
   "DateTime": "2023-10-24T14:32:46.237Z"
  }
 ]
}

This time, the order has successfully gone through. Let’s now confirm if the user’s account balance was correctly adjusted:

Request:

GET /users

Response:

{
 "users": [
  {
   "ID": "6537cb4f8af49a04839d1ac6",
   "Name": "John Doe",
   "Balance": 300
  }
 ]
}

Indeed, now, the user account is also updated. Everyone is happy now.

Gotcha

If you run MongoDB on docker container locally, it will throw an error:

“Error: (IllegalOperation) Transaction numbers are only allowed on a replica set member or mongos”

The reason behind this is that transactions are designed to work with logical sessions, and they’re a bit particular about needing certain background mechanics (like the Oplog), which are exclusively available in the replica set environment. When we run a Docker container, it’s essentially like having a one-person party instead of a lively group.

So, if you attempt to kick off a session on a standalone server, it’s perfectly normal to receive this friendly reminder.

To make transactions work smoothly within a Docker container, you’ll want to set up a MongoDB replica set and follow the steps to get it up and running locally. It’s like ensuring the dance floor is perfectly set for a night of smooth moves!

Suggestion: If you’re looking for a hassle-free experience, why not give the free tier of Mongo Atlas Cloud a try?

Conclusion

In this comprehensive tutorial, we’ve explored a lot of ground. We delved into the potential challenges one might face when dealing with multiple collections. Together, we’ve crafted a robust Go application integrating MongoDB transactions and Gin. Along the way, we established a seamless connection to meticulously defined MongoDB data models. We skillfully implemented handlers for a range of tasks — from placing orders and retrieving them to creating new users.

This application stands as a solid foundation, ready to support even more intricate systems that demand precise atomic operations. Kudos to you for mastering these essential building blocks!

Here is the code for this project: https://github.com/pgaijin66/mongodb-transactions

Please give this article a clap and follow for more interesting content like this. Happy learning.

P.S: If you want to get these article delivered to your mailbox, then please subscribe to my SRE/DevOps Newsletter: ( https://reliabilityengineering.substack.com ).

Better Programming

Multi-Collection Updates and ACID Compliance Using Context and Transactions in MongoDB Using Go

TL;DR

Longer Answer

Understanding Contexts

ACID Compliant Databases

Transactions: Ensuring Data Integrity

Enough Theory, I Need To See Some Code

Prerequisites

Project Structure

Setting Up MongoDB Connection

Initializing MongoDB Client

Defining Data Models

Handling Place Order Request

Handling Order and User Retrieval

Creating a New User

Running the Server

Performing Transactions

Gotcha

Conclusion

Sign up to discover human stories that deepen your understanding of the world.

Free

Membership

Published in Better Programming

Written by Prabesh

Responses (1)