You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
MediaMath's Financial Platform team (FINP) is responsible for building a scalable, stable and secure platform for management of all internal financial data. We serve as the authoritative source of record and support all financial processes (e.g., billing, forecasting, compensation, auditing, etc.) and provide internal clients with access in support of their reporting & analysis needs.
3
+
MediaMath's Financial Platform team (FINP) is responsible for building a scalable, stable and secure platform for management of all internal financial data. We serve as the authoritative source of record and support all financial processes (e.g., billing, forecasting, compensation, auditing, etc.) and provide internal clients with access in support of their reporting and analysis needs.
4
4
5
-
In the last year, the FINP has been renovating its infrastructure moving to a Service Oriented Architecture and joining the trend of Serverless Architecture. In our thinking, using a Serverless Architecture allows for the team to reduce daily operations and spend more time in business-related work.
5
+
In the last year, the FINP has been renovating its infrastructure moving to a service oriented architecture and joining the trend of serverless architecture. In our thinking, using a serverless architecture allows for the team to reduce daily operations and spend more time in business-related work.
6
6
7
-
One of the first services that became alive in this process of renovation was a service that allowed the FINP to run asynchronous queries on a PostgreSQL database, in a safe manner, and save results in S3 and/or send them by email. It was called Preakness. One of the key responsibilities of the FINP is to provide financial reports to team's clients (internal and external), Preakness was an ideal partner to run and send some of those reports. However most of the reports had to run periodically, so we had to find a way to schedule reports running on Preakness and, in general, schedule Preakness jobs.
7
+
One of the first services that was brought to life in this process of renovation was a service that allowed the FINP to run asynchronous queries on a PostgreSQL database, in a safe manner, and save results in S3 and/or send them by email. It was called Preakness. One of the key responsibilities of the FINP is to provide financial reports to team's clients (internal and external), Preakness was an ideal partner to run and send some of those reports. However most of the reports had to run periodically, so we had to find a way to schedule reports running on Preakness and, in general, schedule Preakness jobs.
8
8
9
-
## Potential Solutions
9
+
## **Potential Solutions**
10
10
11
-
### 1. Crontab
11
+
### **1. Crontab**
12
12
13
-
The most simple and straightforward way to schedule task is using Linux [crontab](https://help.ubuntu.com/community/CronHowto), with crontab we could schedule scripts to run periodically and trigger Preakness' jobs. It was simple and feasible but not an option the team was seriously considering: running an instance exclusively to run crontab and trigger the jobs didn't make sense to us. We could re-use or share resources with other services but it hardly would fit in a Service Oriented Architecture and not close at all to a Servessless Architecture.
13
+
The most simple and straightforward way to schedule task is using Linux [crontab](https://help.ubuntu.com/community/CronHowto), with crontab we could schedule scripts to run periodically and trigger Preakness' jobs. It was simple and feasible but not an option the team was seriously considering: running an instance exclusively to run crontab and trigger the jobs didn't make sense to us. We could re-use or share resources with other services but it hardly would fit in a service oriented architecture and not close at all to a servessless architecture.
14
14
15
-
### 2. AWS CloudWatch Events
15
+
### **2. AWS CloudWatch Events**
16
16
17
-
AWS configures events using [CloudWatch](http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html), to invoke other AWS services in a cron-like way. As the team's infrastructure is based on AWS, using AWS services to schedule events made total sense. Preakness also uses Simple Queue Service (SQS) to queue jobs integrated with CloudWatch events. It seemed like a good option, however we found some negative aspects we didn't feel comfortable with:
17
+
AWS configures events using [CloudWatch](http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html), to invoke other AWS services in a cron-like way. As the team's infrastructure is based on AWS, using AWS services to schedule events made total sense. Preakness also uses Simple Queue Service (SQS) to queue jobs integrated with CloudWatch events. It seemed like a good option, however we found some negative aspects we didn't feel comfortable with:
18
18
19
-
* The service itself has some serious [limitations](http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/cloudwatch_limits_cwe.html)
20
-
* Events are attached to AWS services and depend on AWS releases.
21
-
* It is very different from the traditional way to manage and run cron jobs, not user-friendly.
22
-
* Changes in your events imply changes in your AWS infrastructure.
19
+
- The service itself has some serious [limitations](http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/cloudwatch_limits_cwe.html).
20
+
- Events are attached to AWS services and depend on AWS releases.
21
+
- It is very different from the traditional way to manage and run cron jobs, not user-friendly.
22
+
- Changes in your events imply changes in your AWS infrastructure.
23
23
24
-
### 3. Custom solution
24
+
### **3. Custom solution**
25
25
26
-
The third option was to build an in-house solution that satisfying some requisites as:
26
+
The third option was to build an in-house solution that satisfies some requisites as:
27
27
28
-
* Easy and friendly way to schedule jobs: it should allow users (including non-developers) to define and schedule jobs.
29
-
* Reduce maintainability of the service as much as possible: Serverless.
28
+
- Easy and friendly way to schedule jobs: it should allow users (including non-developers) to define and schedule jobs.
29
+
- Reduce maintainability of the service as much as possible: Serverless.
30
30
31
-
We decided to go for the 3rd option. The simple service allowed us to schedule messages to SQS queues by writing few lines in a YAML file. As we were trying to find a solution to schedule jobs for Preakness we realized the potential it had:
31
+
We decided to go for the third option. The simple service allowed us to schedule messages to SQS queues by writing few lines in a YAML file. As we were trying to find a solution to schedule jobs for Preakness we realized the potential it had:
32
32
33
-
* What if it were able to trigger different types of events?
34
-
* Could it be detached from FINP architecture?
33
+
- What if it were able to trigger different types of events?
34
+
- Could it be detached from FINP architecture?
35
35
36
36
That was how LambdaCron was born.
37
37
38
-
LambdaCron is a serverless cron tool. It provides an easy way to run scheduled tasks on the AWS cloud, defined in YAML, managed by a command line tool and using the same syntax for expressions as Linux [crontab](https://help.ubuntu.com/community/CronHowto). LambdaCron fills in the gap by providing a way to manage Serverless cron jobs just like cron. With LambdaCron you define each of your tasks in an independent YAML file.
38
+
LambdaCron is a serverless cron tool. It provides an easy way to run scheduled tasks on the AWS cloud, defined in YAML, managed by a command line tool and using the same syntax for expressions as Linux [crontab](https://help.ubuntu.com/community/CronHowto). LambdaCron fills in the gap by providing a way to manage Serverless cron jobs just like cron. With LambdaCron you define each of your tasks in an independent YAML file.
39
39
40
-
LambdaCron offers 5 different types of tasks:
40
+
LambdaCron offers five different types of tasks:
41
41
42
-
***Queue task**: send a message to AWS SQS queue.
43
-
***Lambda task**: invoke AWS Lambda function.
44
-
***Batch task**: submit AWS Batch job.
45
-
***Athena task**: submit AWS Athena queries.
46
-
***HTTP task**: send HTTP requests (GET & POST).
42
+
-**Queue task**: send a message to AWS SQS queue.
43
+
-**Lambda task**: invoke AWS Lambda function.
44
+
-**Batch task**: submit AWS Batch job.
45
+
-**Athena task**: submit AWS Athena queries.
46
+
-**HTTP task**: send HTTP requests (GET & POST).
47
47
48
-
It is integrated with 4 AWS services and HTTP requests. LamdaCron is easily integrated with other services and it is ready to reach any service available by an API.
48
+
It is integrated with four AWS services and HTTP requests. LamdaCron is easily integrated with other services and it is ready to reach any service available by an API.
49
49
50
-
The tool is production ready but it still has a long way to go. It satisfies the needs of MediaMath's Financial Platform but there is room to increase functionality and improve. The three main lines where we feel it must improve are:
50
+
The tool is production ready but it still has a long way to go. It satisfies the needs of MediaMath's Financial Platform but there is room to increase functionality and improve. The three main lines where we feel it must improve are:
51
51
52
-
***Tasks**: cover all options in current type of tasks and continue adding new types.
53
-
***CLI**: improve and continue adding functionality to the CLI. Tasks should be totally managed from the command line and without login in the AWS console.
54
-
***Templates**: use a templating system to define tasks and be able to include variables in the definition.
52
+
-**Tasks**: cover all options in current type of tasks and continue adding new types.
53
+
-**CLI**: improve and continue adding functionality to the CLI. Tasks should be totally managed from the command line and without login in the AWS console.
54
+
-**Templates**: use a templating system to define tasks and be able to include variables in the definition.
55
55
56
-
## Use Case
56
+
## **Use Case**
57
57
58
-
MediaMath's Financial Platform is really into AWS and loves trying new services. We have been using [AWS Batch](https://aws.amazon.com/batch/) since the release date. But we found it was not possible to schedule AWS Batch jobs directly from the service. One of the simplest ways to schedule AWS Batch jobs is to write a lambda function to trigger a job to schedule the lambda function. It is easy but it implies more code and more infrastructure to maintain.
58
+
MediaMath's Financial Platform is really into AWS and loves trying new services. We have been using [AWS Batch](https://aws.amazon.com/batch/) since the release date. But we found it was not possible to schedule AWS Batch jobs directly from the service. One of the simplest ways to schedule AWS Batch jobs is to write a lambda function to trigger a job to schedule the lambda function. It is easy but it implies more code and more infrastructure to maintain.
59
59
60
-
Here's an example of how to use LambdaCron to trigger an AWS Batch job. This is an AWS Batch job definition used by FINP to run reports:
60
+
Here's an example of how to use LambdaCron to trigger an AWS Batch job. This is an AWS Batch job definition used by FINP to run reports:
61
61
62
62
```json
63
63
{
@@ -88,7 +88,7 @@ Here's an example of how to use LambdaCron to trigger an AWS Batch job. This is
88
88
}
89
89
```
90
90
91
-
This job is used to run weekly and monthly reports so we want to schedule this job to run weekly and/or monthly. A container is running a python program which receives 3 arguments, two of those arguments (Ref::period, Ref::report_type) are parameters that must be passed when submitting the job.
91
+
This job is used to run weekly and monthly reports so we want to schedule this job to run weekly and/or monthly. A container is running a python program which receives three arguments, two of those arguments (Ref::period, Ref::report\_type) are parameters that must be passed when submitting the job.
92
92
93
93
This is a LambdaCron task definition to run the monthly report:
94
94
@@ -105,10 +105,11 @@ task:
105
105
period: '--last-month'
106
106
```
107
107
108
-
The job is scheduled to run at 5 pm on the 5th every month. The **type** of the task in specified inside the key **task** along with parameters required by AWS Batch API to submit a job. After defining the task it is uploaded it to LambdaCron using LambdaCron CLI.
109
108
110
-
Before creating tasks, a LambdaCron environment must be deployed by following the instructions in the [README](https://github.com/mediaMath/lambda-cron) of the project.
109
+
The job is scheduled to run at 5 pm on the fifth of every month. The **type** of the task is specified inside the key **task,** along with parameters required by AWS Batch API to submit a job. After defining the task, it is uploaded to LambdaCron using LambdaCron CLI.
111
110
112
-
LambdaCron CLI also provides commands that can automate development and maintenance tasks. E.G. validate and upload tasks to a LambdaCron environment using Continuous Integration tools. We will follow up with more examples and how to work with LambdaCron in future blog posts.
111
+
Before creating tasks, a LambdaCron environment must be deployed by following the instructions in the [README](https://github.com/mediaMath/lambda-cron) of the project.
113
112
114
-
Any service accessible by an API (including non AWS services) can be reached by LambdaCron so you don't have to rely on additional services features to schedule your tasks and/or write extra code.
113
+
LambdaCron CLI also provides commands that can automate development and maintenance tasks. For example, validate and upload tasks to a LambdaCron environment using Continuous Integration tools. We will follow up with more examples and how to work with LambdaCron in future blog posts.
114
+
115
+
Any service accessible by an API (including non AWS services) can be reached by LambdaCron so you don't have to rely on additional services features to schedule your tasks and/or write extra code.
0 commit comments