Back to all use cases
Financial Services / TradingSupportCloud (AWS)

Functions that fail to start when the subnet runs out of addresses

WM
Wealth Management Firm

Overview

VPC-attached Lambda functions need network interfaces and subnet IP addresses to scale. When a small subnet is shared with other workloads, a burst of concurrent invocations can exhaust available addresses, and the function fails before your code ever runs — which is why the application logs show nothing.

Challenge

During market-open bursts, a share of invocations failed with networking errors and no application log output. The functions were attached to two small subnets shared with other services. Every invocation also opened a fresh database connection at handler scope rather than reusing one, so a burst simultaneously exhausted subnet addresses and the database's connection limit.

Environment

VPC-attached AWS Lambda functions reaching private databases and internal services, with sharp intraday traffic bursts. AceMQ works on the customer's function configuration, networking design, and connection handling — the AWS platform itself is Amazon's.

Approach

AceMQ correlated failed invocations against subnet address utilization and concurrency to confirm exhaustion as the cause rather than an application fault. We then addressed both sides: enough address space and interface capacity to absorb the burst, and connection reuse so the database is not attacked by the same burst.

Solution

  • Correlated invocation failures with subnet IP utilization and concurrency to confirm exhaustion
  • Moved functions to dedicated, appropriately sized subnets across additional availability zones
  • Moved database connection setup outside the handler so warm invocations reuse connections
  • Introduced a connection proxy layer so burst concurrency does not translate into raw database connections
  • Set reserved and provisioned concurrency where burst shape justified it
  • Added alerting on subnet address utilization and function-level throttle and error metrics

Outcome

The market-open failures stopped, and database connection counts during bursts fell dramatically once connections were reused rather than created per invocation.

Technologies

AWS LambdaPostgreSQLRedisDocker

Ready to Get Started?

Whether you need architecture advisory, 24/7 support, or full managed services, AceMQ has the expertise to help.

Contact Us