Web Architecture that Will Satisfy Your Customers in 2021
Well, there are countless ways you can design your next app’s infrastructure, and there are so many good technologies and languages that you might need one more life to learn the better part. So you aren’t able to always have the newest ones in your stack. On other hand, new theories, experiments, new cool features are good for the future, like seeds for the next better technologies. So you don’t want to stick with old but proven technologies either. But when you are responsible for the product’s reliability, you don’t want the peace of the system to mess up the whole product. Big companies have many reasons why they don’t change their existing technologies frequently. You must have at least half of these reasons to choose reliable technologies, the right technologies.
For your next product/project, I offer an architecture where I put together relatively new or really new, but at the same time, proven technologies with their best practices. Here’s how it looks like 👇. (And please note, it can be done using only a single language — Javascript )
I will spread this out one by one being as short as possible, but it would be a long article nevertheless. Also, in a couple of places, it may be a bit harder to understand. Depending on your knowledge and experience you are free to skip any section.
Let’s start from
Back-end
Node.js
Here I use Node.js because it is my favorite runtime, you are free to use the one you’re comfortable with. Node.js works very differently than the others, depending on the case it can be advantageous or the contrary. Huge companies don’t use Node.js for their entire business logic, but they can use it partially to solve a particular problem so Node is extremely popular these days.
Here, the most important part is that you need your back-end runtime to be stateless. This is important for scalability seeks. For example, you shouldn’t store or cache any data on your back-end app, user auth session is the best example of this kind of data, which is inadmissible to do. Instead, you might want to use Redis to store the user sessions, or use JWT. When your back-end server is stateless, you are ready to scale horizontally.
MongoDB
It’s a NoSQL database, the best of its kind. The question is “SQL or NoSQL”. Well, if you are not building the next multi-billion company, you are good to go with any of those two. But it doesn’t mean that you can close the question and stick with the one. Each one has its pros and cons, each one solves a specific problem better than the other. If you are still reading this paragraph, I recommend you to dive into details by googling, at least.
Our Node.js instances are connected to the MongoDB. And additionally, we take advantage of in-memory databases(ex. Redis) and make an interception between backend queries and MongoDB. (Note: You shouldn’t use Redis for every service that asks for data, instead, you might want to use it for data that is queried frequently, and updated infrequently).
Redis
It’s an in-memory key-value database. It’s mostly used to store temporary data (to cache), for faster access. Also for any data that you need to pull off from your runtime app to have a stateless app. By using this kind of technology, we also reduce the number of queries to our MongoDB database.
When our Node.js app asks for data, we first check if we have it cached in Redis, if the answer is yes — Redis returns the data, if no — Redis lets the query to reach MongoDB, and then keeps the copy of the data for the next time (future query which asks for the same data). Our Node.js app can also invalidate the part of the Redis cache when the particular data on MongoDB is updated.
Worker
Let’s pretend that we have an expensive computation to be done in our service. In this case, we never want to load the server we use to accept requests and respond to those. Instead, we allocate additional space and computational power, optimized for such tasks.
GraphQL
GraphQL is a very popular technology for client-server communications. It’s developed by Facebook in 2012 and became open-source in 2015. Since 2015, in a very short period, it became a favorite technology for many developers. Many top companies like Facebook, Github, Airbnb use GrahQL. Instead of traditional REST APIs, wherein the front-end, you interact with already prepared data fields using different endpoints, in GraphQL you have a single endpoint, and you decide which fields do you need for a particular query.
In our architecture, we use GraphQL to serve a fast and flexible endpoint to our client. It also simplifies frontend-backend developer communication and client-side development overall. It’s not a good practice to implement your GraphQL layer coupled with your business logic. So on the server-side, we have a GraphQL server up and running, which holds the GraphQL schemas and resolvers, and communicates with our Node.js server. Now, any GraphQL client can connect to our GraphQL server endpoint, and ask for data. we never want to load the server we use to accept requests and respond to those. Instead, we allocate additional space and computational power, optimized for such tasks.
Amazon Web Services (AWS)
AWS is a leading multi-service platform for almost everything. We use AWS to keep up and running our servers, also for content delivery and data storage, and for a bunch of useful stuff. It lets us have an enterprise wide-infrastructure by writing a miserable quantity of code and paying as we grow.
AWS S3
We use AWS S3 for storing the user-generated content (like avatars, videos, photos), assets, the front-end application, static pages, etc…
We don’t make our s3 buckets public, instead, we use CDN (content delivery network) to serve data from S3. We can also take advantage of S3 events, and trigger Lambdas.
AWS Lambda
AWS Lambda(aka serverless functions) is a computing service. Think of short running server, here’s the lifecycle: something triggers a lambda function, an instance of your lambda function runs immediately, it does the job, it’s gone. And you just pay for the time it was running. Lambda is language-agnostic so you can run node.js, python, ruby, go, java, etc code. You can trigger a Lambda function by an AWS service, for example, API Gateway, and use your lambda as an endpoint controller or business logic server. You can use any AWS service in your lambda function, you can also add custom layers. So it’s a very powerful service.
We use Lambdas for video conversion by creating conversion jobs, and trigger it by s3 bucket when a new video is uploaded. Also for user avatars, to generate different avatar sizes. We record data in our DynamoDB, also send a request to our main server webhooks to transfer some data, or inform about success/fail.
AWS DynamoDB
It’s a NoSQL database system by AWS, it can be used for almost everything (ex. for your business layer).
We use it for our cloud processing data. For example, we convert videos using lambdas and the AWS MediaConvert service, we record in DynamoDB the file analysis data, processing results as well as other useful runtime info. DynamoDB is overall very powerful. If you use AWS, I recommend diving into its features and use cases.
AWS CloudFront
This is the CDN of AWS, which is very fast. It has more than 225 locations on the globe.
We serve the content via CloudFront distributions. It may not be the best nor the cheapest one, but it has many advantages if your infrastructure is built on AWS.
Front-end
React
React is a declarative, efficient, and flexible JavaScript library for building user interfaces, as defined by its creators. Simply put, it allows you to create a fast, simple, and scalable frontend for web applications. React has taken the web development world by storm. In combination with some other libraries, it becomes a killer front-end tool.
We use Create-React-App for client-side rendered apps, and Next.js if we need server-side rendering. Both are very popular and consistent frameworks built on top of React.
Apollo client
“Apollo is a comprehensive state management library for JavaScript that enables you to manage both local and remote data with GraphQL. Use it to fetch, cache, and modify application data, all while automatically updating your UI” — Apollo team
We use it in our React app for reaching our GraphQL endpoint, as well as for data management and caching. It has an in-memory normalized cache, so we don’t need redux for data management at all. It has great dev tools, which improves the developer experience, and reduces the debug time. Apollo client works with React just fine, we’re able to flawlessly reflect the data layer changes on the UI.
Redux
Redux is an independent state management library. It’s mainly used as a single source of truth for the front-end application state. It has middleware plugins, so you can extend your redux, to do debugging in dev tools, to do asynchronous stuff, etc…
We use it just for the UI state, as we use GraphQL, we find Apollo client more preferable for data management and caching. But if you use REST for data fetching, you might want to use Redux for your data management related stuff as well.
That’s all for particular units of our architecture. Now, let’s take a look at the bigger picture.
The big picture
The Client-side
When the user opens our website, he/she requests the index.html file from our CDN. The index.html file contains all necessary assets to get our web app working, particularly our javascript bundle(or chunks), which is also downloaded from the CDN to the user’s browser, and then it’s executed. The bundle itself is our React web app transcompiled to a cross-browser javascript by Babel and bundled by Webpack. After it’s executed, the Apollo GraphQL code is going to fetch the necessary data, and cache it in the user device’s memory, and update the app’s UI (re-render the html code to reflect the new state). That’s the short description of what’s going on under the hood of a client-side rendered React app using Create-React-App.
The Server-side
Now, let’s see the flow of the data being fetched by Apollo Client. Apollo client makes a GraphQL query to our back-end server’s GraphQL layer, which resolves it by communicating with our Node.js server. The Node.js server respectively asks DB for data. Here, Redis comes in and checks if such query is in the cache, and decides whether return the data or pass the query to MongoDB. After DB operations, Node.js receives the data and sends it to the client through the GraphQL server.
Now let’s see our AWS services in action
Pretend a user wants to upload a video. Node.js is nice for handling millions of light requests and business stuff. That’s why we don’t want to load our Node.js server doing uploads. So we upload directly to the S3 bucket but in a secure and consistent way.
Since the user chose the video, our client-side app sends the metadata to our Node.js server. Node.js server takes the metadata, connects with AWS S3, and asks for an endpoint(signed URL) for a particular file upload. As soon as Node.js receives the URL(upload endpoint), it responds to the client’s request. Now we can say that Node.js is done with the upload.
Now, it is the client’s turn to do the job, so the client starts the upload directly to AWS S3 through the signed URL. After the upload is completed, S3 can trigger s a Lambda function to notify the Node.js server that we have a new video. Node.js creates a new record in MongoDB saving the link of the video. Another Lambda can run a video conversion job and generate that video in different qualities, or formats.
So, everything is fast and seamless, everyone is happy.
Conclusion
This architecture is not the most sophisticated one. But this is enough thought out to be enough for most products/businesses for a while. It can be scaled really well and can be easily extended in the context of the features. These two above are very-very important. Depending on the complexity of your product, on the business needs, etc.. you might want to keep it simple and to skip any block of this architecture, and of course, go read about the KISS principle :)
Thank you for reading. I am going to post more stories about different architectures, follow me on Medium.
My github: https://github.com/JeDaVard
