The Sizmek MDX platform Log Files Collection Procedure is a solution for gathering ad impression log files and other log files from our various remote data centers (ECS) into the central data warehouse (EDS).
The collection procedures are comprised of a smart agent residing in each of our bursting/activity servers and acting periodically according to predefined rules.
The agent packages (ZIP) and transfers the log files from the ECS into the EDS. The EDS’s agent extracts the log files and moves them into a staging location to be ready for data warehouse processing.
During the different steps of the log files collection procedures, the following control procedures exist:
- ECS Monitoring: A scheduled task runs every 30 minutes and monitors the agent activity. The monitoring task searches for files with creation dates older than 30 minutes. If such file exists, the monitoring utility writes an error alert, with a proper description, into the event log.
- EDS Monitoring: Very similar in concept to the ECS Monitoring. ECS has 3 phases of monitoring:
3. Inbound Monitoring: Monitors the folder of all incoming log files from remote ECSs. Every 30 minutes, it checks the existence of files older than 30 minutes (if true, it generates an error alert).
4. Post Extraction Monitoring: Monitors the folder of all extracted log files. Runs every 60 minutes and checks existence of files older than 60 minutes (if true, it generates an error alert).
5. Staging Monitoring: Monitors the folder of all log files ready to be processed by data warehouse. Runs every 30 minutes and checks the existence of files older than 30 minutes (if true, it generates an error alert).
Event Log Monitoring
The event log is monitored by a central management system (MOM – Microsoft Operations Manager). Each event log alert is handled by a predefined set of rules. Such rules define how to handle the alert and who should be notified about it.
The central management system (SCOM – Microsoft Operations Manager) generates daily reports about the size of various log folders in the system. This report is checked on a daily basis by the deployment group.
Impression Count Verification
In addition to the logs processing sequence described above, each ECS reports by an Sizmek proprietary protocol about the number of impressions served from that ECS, These numbers are summed up in the BDB as the "BDB daily number of impressions". This number should be identical to the number of served impressions that is aggregated from the log files, on a daily basis, called "RSDWH daily number of impressions".
A daily report lists the difference between the BDB and RSDWH, per placement. This report is checked on a daily basis by the deployment group.