AWS data migration boosts scalability and flexibility for global ad company
The challenge
IAS had decided to gradually migrate their full infrastructure to AWS, including a solution which helps customers track and optimize ad campaigns across publishers and partners such as Facebook. Core to this technology is the ability to collect, process, aggregate, and analyze ad impressions, clicks, views, and other events related to individual ad sessions.
The existing on-premises system for ingesting advertising events had become difficult to support and scale.
It also needed refactoring to run effectively in the cloud, simplify code management and use AWS-native services.
The client engaged Beyondsoft for coding and AWS data-migration support.
The solution
In less than six months, Beyondsoft updated and migrated the IAS solution to AWS. Prior to the migration, Beyondsoft refactored the solution’s entire codebase from top to bottom and implemented a cloud-based ad event ingestion solution to support multiple partners, minimize operational costs and coding, and scale dynamically to accommodate sudden volume spikes.
Beyondsoft reviewed each partner’s application logic to identify shared code and standard locations for partner-specific requirements.
This created a simpler codebase that was easier to maintain and extend to new partners.
The refactoring also improved cloud readiness and enabled greater use of AWS capabilities.
How it works
The comprehensive solution makes the most of AWS functionality while reacting dynamically with speed. To achieve optimal performance, it separates the ad events pipeline into two pieces: a receiver at the front end and a deliverer at the back end.
The front-end pipeline processes advertising events in four stages:
- Receive: A lightweight Node.js service behind Elastic Load Balancing exposes API endpoints for partners.
- Standardise: The application validates incoming calls and converts them into standard advertising-event records.
- Stream: A Kinesis Producer Library process buffers records and sends them to a Kinesis Data Stream, smoothing traffic for downstream processing.
- Scale: Amazon ECS/AWS Fargate and load balancing support automatic scaling and low operational overhead.
On the back end of the Kinesis Data Stream—the deliverer service—a Kinesis Client Library (KCL)-based application enriches ad event records with additional properties derived from external services, and stores the records in the Amazon S3/AWS Glue Catalog-based data lake for further processing. Amazon Cloud Watch collects and surfaces pipeline metrics for dashboard monitoring, and alerts administrators when pipeline errors exceed specific thresholds.
Results
- Reduced operational costs: The streamlined solution in AWS requires fewer resources.
- Increased scalability: IAS can quickly and cost effectively scale to new partners.
- Improved flexibility & performance: IAS leverages the performance and advanced capabilities of AWS.
- Improved integration: The new solution integrates with AWS-based client infrastructure for greater collaboration.
Technologies used
Elastic Load Balancing (ELB), Amazon Elastic Container Service (ECS), AWS Fargate, Amazon Kinesis, Kinesis Data Streams (KDS), Kinesis Producer Library (KPL), Kinesis Client Library (KCL), Amazon S3, AWS Glue, AWS Auto Scaling, Amazon CloudWatch, Amazon DynamoDB, AWS Cloud Development Kit (CDK), AWS CloudFormation
Our experience
Beyondsoft has performed hundreds of data migrations for large enterprise customers. Our certified practitioners have hands-on, best-practice knowledge of all the major platforms. As an AWS Data Migration Competency Partner, Beyondsoft has invested in building deep proficiency on the AWS platform.