Serverless and SQL with Python on AWS

Photo by Caspar Camille Rubin on Unsplash

In one of our latest projects we required a classical SQL database because of the nature of the data and its relations. Whenever possible we are using a servereless approach for our cloud projects. Hence, we decided to build our lambdas with SQL support in an AWS environment. Let me give you some insight on how we did that.

Setup

We started with the following components:

• serverless with AWS Lambda and API Gateway
• MySQL Community Database on AWS RDS

For scaling things really up and down you should consider using AWS Aurora Serverless. Check out this nice article on choosing a database for a serverless application.

In our serverless project we tried to separate the different modules as cleanly as possible so that each module has a clear responsibility. For this showcase we store dogs in the database and access them via a lambda function.

project
  common
    - db.py
  data
    model
      - dog_data_object.py
    - dog_data_access_service.py
  function
    common
      - abstract.py
    - dog.py
  - serverless.yml

Serverless Configuration

We use the serverless.yml file to set the database connection variables as environment variables. This helps us later to run different environment profiles (dev, int, prod, …) and allows us to make local testing more flexible. In this example we just create a dog GET function which will access the database.

Note that we used serverless-python-requirements which makes it easier to manage requirements.

The Connection to the Database

For connecting to our MySQL database we use the powerful python library SQLAlchemy. Alchemy is a wonderful tool to manage the database sessions and helps you in with object-relation mappings.

For that we created a separate python module to manage the database engine, provide sessions and offer an Alchemy JSON encoder. The encoder can be enhanced to address recursive and circular object structure (compare this great post).

The Data Access Module

In order to separate functions and data clearly we introduced a data access module which provides the database objects. For that you need to define your database object which maps to a certain table.

Another data access module provides you the access to your database objects. Note that we pass the SQLAlchemy session as a parameter because the session is created and managed on a function level and could also be used for multiple database access calls.

The Function

Now we want to access the database from our function. For that we created a small abstract handler which is wrapping our function call and which is getting the database session and commits the session after the function has run through successfully.

This abstract handler provides a nice approach in abstracting things like session handling, user management or other general things which apply to your set of functions. Note that we could have also used a decorator approach here.

The actual logic of our function is pretty slim and just concentrates on the actual business logic of fetching the data and returning it.

Summary

Building a SQL based application with a serverless approach is definitely not yet the default in the industry but I believe that it will gain more and more power in the future. The beauty of serverless to abstract from the actual “servers” makes it so much easier as a software engineer in a DevOps world that also more classical applications (which often rely on SQL databases) will eventually consider moving to serverless.

With all the great advantages of serverless there are also two main architectural points that need to be considered when building SQL based lambda functions: Scalability and Separation of Concerns

A serverless approach is a wonderful approach to make your application easily scalable but this does not help you if your database does not scale too. You need to take this point into consideration from the very beginning and find a suitable approach. Obviously the approach in this article is not very scalable but there are many approaches which lead you the way. Here is a nice article on this topic.

Serverless is great to separate your services cleanly and move to a microservice architecture but keep in mind that your data organization needs to represent your microservice architecture. Therefore decide wisely which of your serverless functions are responsible for which part of the data. It does not make sense to have hundreds of functions separating your services but one huge database accessed by all functions without any separation. Separate your data accordingly and have clean data responsibilities for your functions.

I agree to Paul Johnston’s statement that the serverless approach requires a change in mindset and many more examples and experiences to be shared so that it can gain more momentum. Hence, share your opinions and your view of building SQL based applications with a serverless approach.