Node.js: creating a REST server using the Express Framework in a MVC Approach

Hello, it’s been a long time. I am without any free time because of my master’s degree project. In this post I will talk about a piece of a project that I an developing, in which I had to create a REST server with Node.js.

I am a big fan of CakePHP, because it is the first web framework that I used. Because of this, I always try to find similarities with CakePHP when I start developing a project or learning a language. So, I tried to keep the server structure to be the most like CakePHP as possible.

obs: if you are not familiar with the MVC pattern, read this introductory article.

In this project, the most of models methods use the Bluebird package for promises. Furthermore, I chose to use the MySQL database. If you want to use other database, some changes may be needed to the model layer.

The full source code of the server may be downloaded here.

Application Basic Structure

If you are already familiar with the MVC pattern, you will easily understand the directory structure. There is a directory for the controllers and one for the models, but no one for views. I chose to send data from the server only using JSON. If you want to support mode format, like XML, you may use an additional views layer, separated from the controllers.

Before running the server, edit the config/config.js file, with your database configurations and the port to be used by the server. You also may specify the port using the environment variable PORT.

Routes

The Express framework already have a very good routing system, so only a few work was needed here.

Each controller has a addRestRoutes method, which add the basic controller’s routes automatically. These routes map to actions in the Controller class. Each controller of your application must inherit the Controller methods.

The default routes are for an hypothetical BlogsController are

  • GET /blog/1: maps to findById
  • POST /blogs/searches: maps to search
  • POST /blogs: maps to create
  • PUT /blog/1: maps to edit
  • DELETE /blog/1: maps to delete

The search, create and edit actions require some parameters to be passed by the request body.

To the search resource, these parameters must be like the following.

The create and put parameters are only the data to be stored in the database.

If your controller must use some route out of this pattern, you may use the addCustomRoutesmethod.

Controllers

The basic structure of a controller is

The file must be named CarsController.js and to be inside the controllers folder.

Models

The models need some basic configurations in order to use the methods of the Model class.

The main methods inherited from the Model classes are listed below.

Model.find

Do a SELECT query on the database.

Usage

Parameters

  • conditions : array of conditions in the squel format
  • limit : maximum number of records to be returned
  • offset : number of records to be skipped
  • group : SQL group by clause
  • recursive : deep level to look inside the model’s associations

Model.findById

Search for a record with the specified id. Its only parameter is the id to search for. This method returns a rejected promise if there is no record with this id.

Model.save

Create or update an existing record.

Usage

Model.delete

Remove a record from the database. Its only parameter is the id of the record to be removed.

Squel: SQL Generator for Node.js

In this article, I will talk about the squel package, a SQL generator for node.js. To use a SQL generator is very productive, since we don’t need to directly write and debug the queries. Even though I will just show some simple queries, the package is very versatile.

Warning: You may also use squel on the browser, but you shouldn’t run any query in the server that was generated on the client, since it has serious security issues!

Installing and Configuring

In order to install and automatically add squel to your package.json dependencies, run

This package doesn’t need and configuration to be used.

Basic Usage

I took some examples from the official documentation.

Insert

Select

Update

Delete

Using Prepared Statements

Whenever is possible, you should use prepared statements. I wrote an article which shows how to use prepared statements with the `node-mysql2´ package, when you write your queries by yourself.

To do it with the squel package is simple.

And there is much more. It is possible to run subqueries, complex where conditions and generate queries specific to mysql or postgres databases. You really should take a look at the complete documentation!

Caching your views with Redis using the Express Framework

Hi again! On this post, I will show how to use the express-redis-cache package to cache your views, decreasing your page load.

The package is very simple to use, but he is also very complete. I am using it on this blog, and am not missing any important feature. It’s also very efficient, because it uses Redis to store the cache.

Installing and Configuring

To install, include express-redis-cache to your package.json dependencies, or type

I will assume that you are using the 0.1.8 version, which is the most recent at this moment.

So, we must specify the basic configurations of the package.

The prefix option is used to construct the strings that will identify each one of the cached pages. The app object is exported like a module, so we can use the same cache object in other files, with the same Redis connection.

If you is already conected to a Redis server, you can use the option {client : redis_client_object}, omitting host, port and auth_pass. If your Redis server require a password, you need to authenticate the redis client by yourself.

Basic Usage

The cache.route() middleware must be used on every route that should be cached.

This will ask to cache to check if there is some cache named APP_ NAME + '/'. If not, thehome() function will be executed, and ‘Welcome to my app’ will be automatically cached, with the APP_NAME + '/' name. While it doesn’t expire, the home() function will no longer be called.

Changing the Entries Name

You can change the name of a entry using

or

or

Choosing to Cache (or not) a Page

You can chose not to use the cached answer on the fly.

Ovewriting the expire option

To change the expiration time of a single entry, do

Excluding Cache Entries

You can exclude cache entries, even if they haven’t expired yet.

If you have pagination at your homepage, with each page saved on entries like home.page.0,home.page.1, and so on, you can exclude all these entries with a single call to del() function.

Debugging

To listen the various debug messages sent by the package,

In medium/large applications, your page load will decrease considerably if you cache your views. However, you must be careful to clear the cache whenever your views are updated. You don’t want an old HTML including a updated javascript, do you?

Generating a Blog Sitemap with Node.js using the Express Framework

In this article I will talk (superficially) about what sitemaps are, and how to generate them with an existing Node.js package. Even though the practical examples are focused on blog applications, they may easily be extended to a more general website. The sitemap will be cached by 24 hours (the time may easily be changed).

I chose to use the sitemap package, which is not specific to Express, but is very easy to use, and has an interesting cache system. You may take a look at the express-sitemap package, specific to Express, but the only advantage that I found in it is that it can automatically add static pages of your app object to the sitemap. If you have lots of static pages, this may be useful for you. For a blog, I haven’t found any reason to use it instead of the more generalsitemap package.

If you know the basic about sitemaps, you may skip the first section of this article.

An Introduction to Sitemaps

Sitemap is a protocol made for helping search engines on pages indexing. It may be a single text file, on which each line is the URL of a page you want it to be indexed.

Naturally, if you don’t include some URL on your sitemap, this page may still be indexed. If you don’t want some page to be indexed, read about robots.txt. Likewise, if you add a page on the sitemap, it may not be displayed on the first page of any search engine.

Sitemaps are just tools to help the search engines to find every page of your site. Optionally, you may add a relative priority to every page of your site, if you think that some pages are more important than others. This priority is relative only to pages of your own site, and doesn’t have any effect to make your site be better ranked than other sites.

The most common sitemap format is the XML.

Here,the http://www.example.com/ was updated for the last time on 01-01-2015, is expected to be updated monthly, and has a 0.8 priority (in a scale of 0 to 1). The default priority is 0.5.

Installing and Configuring the Sitemap Package

To install the package, include sitemap to the dependencies of your package.json file, or use the npm.

Now create a routessitemap.js file and add it to your app object.

So, you can generate the sitemap

Visit yoursite.com/sitemap.xml, and there it is!

Including your Posts on the Sitemap

Let’s get the URLs of yours posts, so we can include the on the sitemap. You don’t need to addevery page of your site to the sitemap, only the ones which offers some new content. Pagination or search pages, usually, don’t need to be included.

For this reason, we will add to the sitemap only the URLs of the posts.

The sm.toXML() method checks if the XML needs to be generated again, or if the cache is still valid. If your sitemap have lots of pages, this will result on a great performance gain.

If you prefer to use some package specific to Express, you may use the express-sitemap with only a few modifications. The most important is that the express-sitemap doesn’t have any type of cache. To avoid searching every post from the database, and creating the XML every time the sitemap is requested, you can add a cron job to generate the XML at regular intervals, or save the last time a sitemap was generated in a variable, and check for the expiration by yourself.

Using Node.js with MySQL

Hi there! In this post, we shall talk about the use of MySQL database with Node.js. I will use the node-mysql2 driver. Unfortunately, the driver doesn’t support promises. If you want to use promises instead of callbacks, you may use bluebird promisification.

Installation

To install node-mysql, runs the command

You may also install adding the mysql2 package to the dependencies list of yourpackage.json file. To more details, check package.json;

Configuration

Create a config/database.js file, with the convenient adjustments.

The supportBigNumbers options must be true if any of your tables has a BIGINT or DECIMALcolumn.

Check the full list with the possible configurations here.

Basic Usage

In order to stablish a database connection, use the following code.

So, you can easily run queries

The end() method close the connection, waiting for every pending queries to be executed before this.

Prepared Statements

To avoid SQL Injection attacks, you should use prepared statements whenever you make use of user provided data. Furthermore, prepared statements runs very quickly when similar queries are executed successively.

You can also prepare the statement by yourself.

Connection Pool

The driver also supports connection pool. To do so, add to your config/database.js file the line

This limits the maximum of simultaneous connections to the database.

Now, we may get connections from the pool.

Transactions

It’s very simple to use transactions.

No error handling has been done for readability reasons. Note that this functions just runs theSTART TRANSACTION, COMMIT and ROLLBACK commands, and some queries may cause implicit commits.

Using Promises

Unfortunately this package doesn’t natively support promises. However, you may use thepromisify() and promisifyAll of the bluebird package.

Jade: Creating reusable blocks witn Mixins

he benefits of modularity have been widely discussed on software engineering, are not the focus of this article. Here, i am only concerned with the implementation aspects. If you are interested, check also the inheritance section on the Jade official documentation.

The examples used on this article are taken from the official documentation.

Mixins

Mixins are reusable code blocks, which help us to make our views clean and organized. Firstly, let’s see a simple example which illustrates the mixins’ declaration and usage.

The following Jade code

generates the HTML

The list mixin may be used on any jade file, since the file on which is is declared has been properly included.

Arguments

Now, let’s see a more useful example, on which the mixin takes arguments.

produces the HTML

Blocks

Also, mixins may receive blocks, in order to produce more customized HTML.

will generate the HTML

Named Attributes

You may use named attributes too.

producing the HTML

The named attributes are automatically escaped when calling the function. So, the must be used on the mixin with the ! operator, or the attribute will be escaped twice.

Unknown Arguments Number

Lastly, mixins may receive an unknown number of arguments. The following Jade code

produces the HTML

An Introduction to the Express Framework

Express is a Node.js framework to web applications creation. It is focused mainly on routes which redirect each request to the proper handler function. A request may be handled by one or more functions, called middlewares.

The middlewares may parse POST data and turn them on a format that is easier to be used by others middlewares, e.g. In this post I’ll explain with more details what middlewares are and how they work. In the end, I will list some middlewares that I think should be useful.

Installation

First, execute the command

This will install the express own project generator. So, execute

This will creates a directory caller myapp, with a pre-configured application to use all the express’ basic functionalities. Furthermore, all the directory tree will also be created, including a public folder to your assets.

To start the server, type

Note that the application’s basic code is on app.js.

Middlewares

Every connection made with the server must be handled by one or more middleware, from a simple request to get the application favicon until a request on some resource at a RESTserver.

Each middleware must register itself on an express object, usually named app, saying which kind of request he wants to handle. Optionally, the middleware may choose to handle every requests. This registration is done using the app.usemethod (read the docs here).

The easiest way of use the use method is

This piece of code says to the app object that every requisition done to path should be handled by the middleware function. The path parameter is optional, and, if omitted, is assumed to be ‘/’. This will make the middleware function to be executed on all requests done with the server.

The req and res parameters have, respectively, the data of the request made with the server, and response that will be sent to the client. next is a function which says to the app object that other middlewares should be executed to complete the request handling.

Let’s see some examples:

The code above search for users with req.params.id id, and return it’s data on JSON format. We may improve it to accept only numerical ids doing app.use('/users/:id([0-9]{1,})').

To show an example of the use of two middlewares handling the same request, look at the next example, which is used on the blog to control the access to the admin area.

app.js

routes/admin.js

This example also shows how we can split the routes on different files, improving the code organization. The router.all function says that every king of requisition must be handled by the middleware isLogged'. If the user who is trying to access the admin area is logged, thenext` function is executed, and the other middlewares may provide the requested resources. Otherwise, he is redirected to the login page.

The principal middlewares i used on the blog are:

  1. body-parser – parser the data on request body. I use it to simplify the access to the POST and PUT data;
  2. less-middleware – This middleware is a LESS compiler that handles with every request to .css files, searches for the correspondent .less file and compile it. In order to prevent the repeated compilation of the same file, the middleware looks the last modification date of the .css files and compares with the last modification date of the .less files.
  3. express-morgan – logs all requests, on the console or on a specified file.
  4. express-session -like the name says, this middleware is responsible for handling the user sessions. He don’t store the sessions alone, but needs receive a storer on the its configuration. Each middleware executed after this will have direct access to the user session by the req.session object.

I’ll explain with more details each one of these middlewares in a future post.

Authentication with Node.js and Express

The authentication is a very important part of almost any application. This post shows how to authenticate users on a Node.js server, using the Express, and storing the sessions with Redis.

Configuring the Session

Before we authenticate users, we must properly setup the Express sessions. Install theexpress_session package running the following command

After that, you must setup the installed package.

This way, the user session will be available on the req.session object.

Login

So, the login routine is shown below.

To access the POST data like above, you must use the body-parser package, and setup it

on the Express initialization routine.

Protecting the User Password

Much was already discussed about why encrypt the user passwords on the database. In short, if a malicious user gains access to the database, he, at least, will have a lot of work to recovery the original passwords if they were encrypted. This is important, since many people use the same password on every service.

To encrypt the passwords, I used the bcrypt package.

Also, is a good practice to use different salt strings for every user. This is due the fact that most people use common passwords, like 12345678 or password. So, if most of the users use some of these common passwords and a malicious user gains access to the database, it is not much hard to look for repeated passwords and test them against the most common used passwords.

In order to use bcrypt on Windows, i had to install the Visual Studio. You may download the free version here.

So, my login routine may be summarized on these steps:

  1. Check if there is any user with the informed e-mail;
  2. Get the salt string of this user;
  3. Encrypt the informed password with this salt;
  4. Check if the encrypted password matches with the one stored on database.

The implementation of these steps is

 

An Introduction to Node.js

Node.js has some quite singular characteristics, but one of the most importants is that all the connections made to the server share the same thread.

This means that, if the server is handling with a thousand connections simultaneously, a single thread is responsible to manage all of them.

This way, the server may handle with a high number of connections simultaneously, using no much more RAM than it uses when handling with a single connection.

One with a good knowledge about threads may see this brings some limitations.

As the server manage all its connections with a single thread, if one of them needs to perform some slow operation, like heavy mathematical operations or to resize big images, letting the server busy, no other connections will be answered until this operation ends. Therefore, every request must be answered very quickly!

I/O operations, like database or external resources access, are all asynchronous, like conventional Ajax requisitions. This kind of operations are handled by separated threads, but once the required data is available, they are sent to the main thread by a callback function. So, while a thread get the required data, the main thread may handle with other connections.

Furthermore, Node.js is ideal to real-time applications, like a chat. When one sends a message, it’s expected it to be delivered at the same time to whoever it was sent.

A programmer with low experience could think doing it sending lots of ajax request to the server, asking for new messages. But, with just a few users, it will be no much different than a DDOS attack.

In order to solve this problem, instead of establishing constantly new connections between every client and the server, only one connection is established (per client), which is kept open until the application ends. Using this connection, client and server may exchange messages very quickly.

On the traditional way, to send small data, it takes much longer to establish a new TCP connection than to send the data itself. If you need more detailed information, read more about HTTP and TCP.

On the other hand, using persistent connections, the connection is established only once, and data can be sent by it whenever is necessary.

I will not write any code on this post, because it is already too big, but I hope to write some lines of code soon :)