This post is part of the StormScribe Devlog series. You can find the previous episode here.
If you have ever built something larger than a landing page, there was probably a moment when you developed the need to store data somewhere. At this point, chances are that, sooner or later, you have concluded that you need a database. You might also have discovered that you have multiple options, which can be overwhelming. Not to speak about the various libraries that you can use to interact with them.
The variety of options is generally good but can be frightening when starting. It also means you must learn the usage of multiple libraries in case you want to switch to a different database for whatever reason. This is where ORMs come into the picture.
What is an ORM?
ORM (short for Object Relational Mapping) is a technique used in software development to map objects to (relational) database tables. This way, you can query your database using the well-known syntax of your chosen programming language without having to write any SQL queries. Here's an example:
Suppose you need to get the user with the email johndoe@example.com from your database. To achieve this, you would probably write something like this in SQL (assuming you have a table called users, which has a column called email):
SELECT * FROM users WHERE email = 'johndoe@example.com';
Now, if you are using MongoDB, the syntax quickly changes to something else:
const client = await MongoClient.connect('mongodb://localhost:27017');
const db = client.db('my-database');
const users = db.collection('users');
const user = await users.findOne({ email: 'johndoe@example.com' });
This is quite the difference, isn't it? And this is just one of the simplest queries you can imagine.
Now, let's see what happens after introducing an Object Relational Mapper. My ORM of choice was Prisma, so I will use it throughout this post. Here's what the same query looks like with Prisma:
const user = await prisma.user.findUnique(
{ where: { email: 'johndoe@example.com' } }
);
So what's the gain here? Why would I recommend learning a third syntax when you already know SQL and Mongo? The short answer is that it's enough to learn only this one, and you can use it with any database that Prisma supports.
The long answer is that it's not only about the syntax, although that's a huge benefit. If you are using an ORM, you can switch to a different database without changing your application's code. Apart from a minor configuration that you must do in the ORM's config, you don't have to touch your code at all.
On the other hand, using an ORM also gives you significant security. While you cannot eliminate the possibility of SQL or NoSQL injection, you can greatly reduce its risk. But injections are worth a separate post, and they are out of the scope of this one.
A third benefit is that you don't have to worry about setting up the connection to the database, which is also database-specific. You save time and code.
Why Prisma?
So why did I choose Prisma? It isn't the only ORM out there, although it's among the most popular. The reason is simple: I needed an ORM supporting SQL and NoSQL databases. And since Prisma supports MongoDB, the choice was obvious.
When I started, I was pretty new to both Prisma and MongoDB. Specifically, I was beginning with Mongo in itself, without any ORM. As StormScribe evolved, I started to define more and more relations between my Mongo collections, which isn't really what Mongo was designed for. I also wanted to keep my database schema centralized to see what relations I have defined quickly. Prisma helped me with both of these.
While Mongo is not a relational database, Prisma can sort of turn it into one. By passing every request through Prisma, the ORM layer will check for relations and throw an error if it finds something is missing.
"Why did you start with Mongo in the first place?"
You might ask. The answer is that I wanted to keep the project's costs as low as possible, and Mongo is completely free for small projects. However, I will have to migrate to SQL sooner or later because it is the solution that was designed for relational databases. Now that I am using Prisma, this process won't take more than a few minutes if everything goes right. You will hear about it in a future episode when I do it.
Keep it simple
I had lots of problems with Mongo in the beginning. I found the following code snippet somewhere on the internet and have used it for a long time:
import { MongoClient } from 'mongodb';
const uri: string = process.env.MONGODB_URI!;
const options = {};
let client: MongoClient;
let clientPromise: Promise<MongoClient>;
const wrapGlobal = global as any;
if (!process.env.MONGODB_URI) {
throw new Error('Please add your Mongo URI to .env');
}
if (process.env.NODE_ENV === 'development') {
if (!wrapGlobal._mongoClientPromise) {
client = new MongoClient(uri, options);
wrapGlobal._mongoClientPromise = client.connect();
}
clientPromise = wrapGlobal._mongoClientPromise;
} else {
client = new MongoClient(uri, options);
clientPromise = client.connect();
}
export default clientPromise;
This code differentiates between development and production environments and tries to reuse MongoDB connections in development mode. It worked but caused many problems in testing and closing connections, which wasn't trivial either. Now compare it to the way Prisma handles it:
import { PrismaClient } from '@prisma/client';
export const prisma = new PrismaClient();
That's it. You can import the client anywhere, and it will simply work. Closing the connection is also straightforward:
await prisma.$disconnect();
There might be better ways to handle the Mongo client setup, but I found Prisma much easier to understand and use. It also automagically solved all of the problems I had during testing.
Prisma setup
To keep the fight fair, I have to mention that Prisma also requires a bit of setup at the start. You must define your database schema in a file called schema.prisma, and you have to run a command after each change in the schema to generate an up-to-date client. Here's a simplified version of my schema:
datasource db {
provider = "mongodb"
url = env("DATABASE_URL")
}
generator client {
provider = "prisma-client-js"
}
model User {
@@map("users")
id String @id @default(auto()) @map("_id") @db.ObjectId
name String
email String @unique
linkedInCredentials LinkedInCredential[]
}
model LinkedInCredential {
@@map("linkedInCredentials")
id String @id @map("_id")
name String
...
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
userId String @db.ObjectId
}
My real schema is much more complex, but this fragment is enough to show how it works. At the top, I define the provider and the database URL. Then, I define a generator that will be used when generating the updated Prisma client. Finally, I define my models. The capitalized model name is used in TypeScript, while the lowercase one in @@map is used in the database.
When the schema changes, you must run a couple of commands to update the client and the database.
npx prisma generate
npx prisma db push
The first one generates a new client for TypeScript. If you don't run this command, you won't have access to the new fields and models in your code. For example, if you just added the TwitterCredential model to the schema, you cannot write the following until you generate a new client:
const twitterCredential = await prisma.twitterCredential.findUnique(
{ where: { id: 'some-id' } }
);
The second command applies your changes to the database. Generally, prisma db push is only used in development and prototyping because this command applies the changes to the database in one go without any migrations. Migrations track how the schema changes over time and are recommended to use when it happens. Think of migrations like git commits but for your database schema.
Additionally, prisma migrate dev warns you if any of your migration steps would result in data loss. However, I didn't have a choice, because MongoDB only supports prisma db push.
Now that you know how to use Prisma to interact with your database, let's face the next challenge that usually emerges during local development: dependencies. Everyone hates them, especially if they are working together with others where there are as many operating systems as colleagues.
But even if you are working alone but use multiple computers, setting up the same environment is often cumbersome.
Docker
I am a huge fan of containerizing everything for local development. It might sound overkill for some projects, but trust me, it's worth it. Okay, you don't need it for a landing page or a blog like this one, but as soon as you start to add more services like a backend or a database, Docker quickly starts to become more and more valuable.
I often switch between my laptop and desktop and hate setting up the same development environment twice. Not to speak about the fact that, more often than not, the same configuration doesn't work on both machines. Even if they are running the same OS. Docker eliminates this problem. The only dependency I have to install on my devices is Docker itself.
Then, I will build the container only once and start it. Installing the correct Node version, the MongoDB local server and the Mongo Explorer is unnecessary. No need for Next.js either on the frontend. Everything is in the image from which I build the container. Also, it will work in the same way on Windows, Linux, and macOS.
Now that you know how to set up your local environment, let's return to Prisma because there are other reasons why Docker is extremely useful in my setup.
Transactions in MongoDB
There is one caveat to using MongoDB with Prisma. When you introduce Prisma, it tries to use MongoDB with transactions by default. When you interact with a relational database, it is often advised to use transactions to avoid potential errors. Transactions are multiple database operations in a sequence that will only succeed if every operation within the transaction has been executed correctly. [1] However, this is not possible with a simple MongoDB instance. You have to use something called a replica set.
In a nutshell, a replica set is a group of MongoDB instances where each instance contains the same data. The point of using replica sets is to ensure availability and mitigate data loss. If one instance fails for some reason, the others will take over the load and handle requests.
It is advised to use at least three instances in a replica set, but in a development environment, you are free to use only one. It isn't much harder than starting your Mongo instance with --relpSet "rs0", but with Docker, you don't have to care about it either.
Here's is the content of my docker-compose.yml I am using for the backend, the MongoDB instance with replica sets enabled, and the MongoDB explorer together:
version: '3.4'
services:
stormscribe-backend:
image: stormscribe-backend
build: ./
env_file:
- .env
working_dir: /usr/src/app
volumes:
- ./:/usr/src/app/
- node_modules:/usr/src/app/node_modules
ports:
- 8080:8080
networks:
- stormscribe
command: npm run start
depends_on:
- mongo1
- mongo2
mongo1:
image: mongo:6.0.5
restart: always
expose:
- 27017
ports:
- 27017:27017
networks:
- stormscribe
- mongocluster
volumes:
- mongo1_data:/data/db
command: mongod --replSet rs
mongo2:
image: mongo:6.0.5
restart: always
expose:
- 27017
ports:
- 27018:27017
networks:
- stormscribe
- mongocluster
volumes:
- mongo2_data:/data/db
command: mongod --replSet rs
mongoinit:
image: mongo
restart: on-failure
networks:
- stormscribe
- mongocluster
depends_on:
- mongo1
- mongo2
command: >
mongosh --host mongo1:27017 --eval
'
rs.initiate({
_id: "rs",
members: [
{_id: 0, host: "mongo1:27017"},
{_id: 1, host: "mongo2:27017"},
]
})
'
mongo-express:
image: mongo-express:1.0.0-alpha.4
restart: always
ports:
- 8081:8081
environment:
ME_CONFIG_MONGODB_URL: mongodb://mongo1:27017,mongo2:27017?replicaSet=rs&readPreference=primary&ssl=false/
networks:
- stormscribe
- mongocluster
depends_on:
- mongo1
- mongo2
volumes:
node_modules:
mongo1_data:
mongo2_data:
networks:
stormscribe:
external: true
mongocluster:
I am using two replicas here. I started with three because I didn't know I could use less than three when I was setting the whole thing up. As an experiment, I downgraded it to two, and I just recently read about the fact that I could use only one when I started writing this episode. I am keeping it at two for now in case you need it in your setup.
This docker-compose.yml contains five services. Covering every line would be a bit too much for this post, but mentioning each one briefly is worth a try.
The first service is the backend itself, which is an Express server. The second and third are the two MongoDB instances. The fourth one is a helper service, which runs only once at each startup if everything goes right. It initializes the replica set and its members. Finally, the last service is the Mongo Explorer, a web interface for MongoDB.
And here comes the beauty of Docker: I only have to run two commands, and the whole backend and database are running.
docker-compose build
docker-compose up -d
I build the backend with the first one because the image of the first service is defined in a separate Dockerfile. Then, I start every service together in the following command. The first attempt takes a bit longer since Docker has to download the other images from the Docker Hub. When they are also built, everything starts. And that's it.
There is no need to worry about starting the Express server, the Mongo instances, and the Mongo Explorer. No need to download any other software either, except Docker.
I left out one essential piece of information from this part: how the DATABASE_URL environment variable looks in this setup. You don't see it in the compose file because I follow a pattern of putting all the environment variables in a .env file and passing it to the compose file in the env_file parameter. The DATABASE_URL looks like this:
DATABASE_URL=mongodb://mongo1:27017,mongo2:27017/dev?replicaSet=rs&readPreference=primary
If you look closely, you can find it in the mongo-express service, but it isn't trivial that you can use the same URL in the backend, so here it is.
When you set any variable in .env, Docker will read and expose them at each startup inside the container so your application can access it. If you look at the URL, you can see that it contains both replica set members, mongo1, and mongo2. When using docker-compose, you can refer to your services by their name because Docker will assign those names to the containers as hostnames.
And that's it. You now have a working backend with a database and a database explorer. You can start everything with a single command, and accessing your records from the code is also straightforward and database-agnostic.
Conclusion
Introducing Prisma was a challenge when I had already written a significant amount of code using the official MongoDB library. But it delivered countless benefits. The first lesson to remember here is that you should never skip planning if you are building a large-scale project. It might look like a waste of time at first, but it will save you days of work later.
Introducing an ORM increased the security of my application since Prisma does not pass the user input directly to the database. It still enables me to write raw commands, but doing so would be a user error and not a limitation of the ORM.
Another note is that you can turn MongoDB into something close to a relational database. I know that MongoDB was not made for this, and you should always use a database designed for the purpose you are using it for. Otherwise, you will quickly run into problems. My only reason to use Mongo was that you can start a Proof of Concept for Free with it. As soon as my app will grow to a specific size, I will migrate to a relational database. And since I introduced Prisma, it won't take longer than a few minutes.
Finally, it is worth remembering that a small budget can make people very creative, but this should never come at the cost of security and not following well-established best practices. The goal is to balance cost-effectiveness, usability, and security.