System Maintenance and Advanced Workflows Running Scheduled Maintenance Automations

Keeping your system tables clean and performant requires routine maintenance. This guide covers the automated background workflows that manage data retention, specifically focusing on daily cleanups for Order Logs, Title Code Comparisons, and Window Management items.

By automating these processes, your system automatically purges outdated records without requiring manual intervention.

Maintenance Overview

The maintenance system relies on a two-step lifecycle for most records:

  1. Stamping an expiration date when a new record is created.

  2. Running scheduled sweeps to find and delete records that have reached their expiration date.

flowchart TD
    A["New Record Created"] --> B["Automation calculates retention date"]
    B --> C["Record updated with 'Delete on' date"]
    C --> D{"Scheduled Nightly Trigger"}
    D -->|"Matches 'Delete on' date"| E["Queue for deletion"]
    E --> F["Loop and delete (Max 500)"]

Retention Policies at a Glance

Target TableRetention PeriodCleanup Schedule
Order Logger14 daysDaily at 1:15 AM (EST)
Title Code Comparison14 daysDaily at 1:15 AM (EST)
Window Management1 dayEvery 3 days at 2:30 AM (EST)

All scheduled times are configured for the America/New_York (EST/EDT) timezone.

How Expiration Dates are Set

To ensure records are deleted at the right time, automations listen for new entries and immediately stamp them with a future deletion date.

Data Retention Logic

When a new record is added to the Order Logger or Title Code Comparison tables, a background automation instantly calculates a date 14 days in the future and saves it to the record's Delete on field.

Scheduled Cleanup Workflows

The actual deletion of records is handled by scheduled tasks that run during off-peak hours.

Daily 1:15 AM Cleanups

Every day (including weekends) at 1:15 AM, two separate automations run to clean up the Order Logger and Title Code Comparison tables.

Here is exactly how the daily cleanup process works:

  1. 1

    Trigger the schedule

    The automation wakes up at 1:15 AM EST.

  2. 2

    Format the current date

    The system formats today's date into a standard MM/DD/YYYY format.

  3. 3

    Search for expired records

    The automation searches the target table for any records where the Delete on field exactly matches today's formatted date.

  4. 4

    Filter and verify

    If no records are found, the automation stops. If expired records exist, it continues to the next step.

  5. 5

    Batch delete

    The system loops through the found records and permanently deletes them from the table.

Batch Limits: The deletion loop is capped at 500 records per run to prevent system timeouts. If you consistently generate more than 500 disposable records per day in a single table, you may need to increase the frequency of these automations.

Window Management Cleanup

The Window Management cleanup operates on a slightly different schedule and logic. Instead of relying on a pre-calculated Delete on date, it evaluates records dynamically based on when they were added.

  • Schedule: Runs every 3 days at 2:30 AM.

  • Logic: The automation takes the current date, subtracts 1 day, and searches the [Table] HDPHD sync - update Window management table.

  • Action: It finds any records where the First Added date is less than or equal to that calculated date and deletes up to 500 of them.

Troubleshooting

Why are old records still showing up in my tables?

There are a few reasons this might happen:

  1. Batch Limits: If more than 500 records expired on the same day, the automation will only delete the first 500. The rest will remain until manually cleared or until the automation is adjusted to run more frequently.

  2. Missing 'Delete on' Dates: If the automation that stamps the 14-day expiration date was paused when the record was created, the record won't be picked up by the daily cleanup.

Can I recover deleted records?

No. Once the scheduled maintenance automation deletes a record from the table, it is permanently removed to free up system resources.