Resilient Backend: Signals management

Overview

In order to build a resilient nodejs backend application there is a special chapter for managing termination signals. Here there are many reasons why Kubernetes might terminate a perfectly healthy container.

• Draining nodes
• Rolling deploys
• Scaling down operations
• Eviction due to resource constraints
• Manual pod deletion

The termination lifecycle in kubernetes go through several steps:

1. Pod is set to the “Terminating” State and removed from the endpoints list of all Services (At this point, the pod stops getting new traffic)
2. preStop Hook is executed (usually used for gracefull shutdown when SIGTERM is or can’t be managed in the application itself)
3. SIGTERM signal is sent to the pod (This is where listener for managing SIGTERM should be managing e.g. stopping any long-lived connections, etc)
4. Kubernetes waits for a grace period (default of 30s)
5. SIGKILL signal is sent to pod, and the pod is removed

Signal management

SIGTERM (Signal 15): This signal is the default signal sent to a process for termination.

SIGINT (Signal 2): This signal is generated when a user interrupts the application process using the keyboard shortcut Ctrl+C. Although Kubernetes does not directly send SIGINT to terminate a pod, it can be used during local development or testing.

Managing termination signals in a nestjs node application

Create new service for managing graceful shutdown

import { Inject, Injectable, OnApplicationShutdown } from '@nestjs/common';
import { UserRepository } from 'src/controllers/db/user.repository';

@Injectable()
export class GracefulShutdownService implements OnApplicationShutdown {
  @Inject(UserRepository) private userRepository: UserRepository;

  async onApplicationShutdown(signal?: string) {
    console.log(`Calling ${signal} onApplicationShutdown`);
    await this.userRepository.shutdown();
    process.exit(0);
  }
}

Register the service in the corresponding module as a provider

import { Module } from '@nestjs/common';
import { UserRepository } from 'src/controllers/db/user.repository';
import { UserController } from 'src/controllers/http/user.controller';
import { UserService } from 'src/core/use-cases/user.service';
import { GracefulShutdownService } from './shutdown.service';

@Module({
  controllers: [UserController],
  providers: [
    UserRepository,
    {
      provide: 'IUserRepository',
      useClass: UserRepository,
    },
    UserService,
    GracefulShutdownService
  ],
})
export class AppModule {}

In your main.ts file, add the following code to gracefully handle shutdown signals

import { NestFactory } from '@nestjs/core';
import { AppModule } from './framework/app.module';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  app.enableShutdownHooks();
  await app.listen(3000);
}
bootstrap();

See full example repository here

By following these steps, your NestJS application with TypeORM will be able to handle graceful shutdowns by properly closing the database connection, allowing for a clean and controlled termination process.

Conclusion

Kubernetes can terminate pods for a great variety of reasons, and making sure your application handles these terminations gracefully is core to creating a stable system and providing a great user experience.