The user is asking about a scenario where after deployment, both cron and a systemd timer register the same job, creating a double-fire risk. They want me to: 1. Verify the double-fire risk 2. Resolve it without breaking either mechanism 3. State which mechanism to remove and why The context seems to be a continuation of a deployment/DevOps conversation ("Same, plus:" suggests this is a follow-up to a previous scenario, though I don't have that context). The question is technical and infrastructure-focused. Let me think through this carefully: ## Verification of Double-Fire Risk First, how would you verify that both cron and systemd timer are scheduling the same job? **Cron side:** - Check user crontabs: `crontab -l`, `crontab -l -u ` - Check system cron files: `/etc/crontab`, `/etc/cron.d/*`, `/etc/cron.hourly/`, `/etc/cron.daily/`, etc. - Verify the schedule: `grep -r "myscript" /etc/cron* /var/spool/cron/` **Systemd timer side:** - `systemctl list-timers --all` - `systemctl list-unit-files --type=timer` - Check timer unit: `systemctl cat myjob.timer` - Check when next fires: `systemctl list-timers` shows NEXT/ELAPSED/PASSED columns - Analyze the calendar spec: `systemd-analyze calendar ` **Verification steps:** 1. Enumerate all schedulers: - `crontab -l` (root) - `crontab -l -u ` - `grep -rn 'myscript' /etc/cron*` - `systemctl list-timers --all | grep -i myjob` - `systemctl list-unit-files | grep -i myjob` 2. Confirm both point to the same executable/script/entrypoint: - Compare the command in cron entry vs `ExecStart` in the service unit - Resolve paths, users, environment differences — a subtle trap: they might *look* different (different paths, wrappers) but hit the same target, or conversely look the same but differ in env. 3. Check effective schedules: - `systemd-analyze calendar "..."` to resolve the timer's OnCalendar - Cron fields are straightforward but beware `/etc/cron.d` with different users, or `/etc/cron.daily` semantics (run-parts, anacron) - Timezone differences! Cron uses system TZ (unless CRON_TZ), systemd timers use systemd's default TZ unless specified. A TZ mismatch could make them fire at different times — which would be a different bug (double-fire at different times or one firing when unintended). Actually if TZs differ, they may fire at different times, still double-firing the job but not simultaneously. 4. Historical evidence: - Journal: `journalctl -u myjob.service --since ...` - Cron logs: `/var/log/cron`, `journalctl -u cron` / `grep CROND /var/log/messages` - Look for interleaved executions at the same timestamp on recent days. - Application-side dedupe logs or DB records showing duplicate runs. 5. Runtime confirmation: - Temporarily instrument: add a wrapper that logs timestamp+scheduler source (e.g., env var), or check `ps -ef | grep myscript` at scheduled time. - systemd-cron? Note: some distros ship `cron` translations of systemd timers or vice versa (e.g., `systemd-cron` package converts /etc/cron.d into transient timers — that could be the root cause: the deploy tooling installed both because the distro's cron implementation is systemd-backed... actually systemd-cron generates timers FROM cron files, so they wouldn't double-fire; but if a package ships both a cron file and a timer unit intentionally with guards, removing one guard causes doubles). Actually, an important nuance: many packages historically ship both a cron entry and a timer (e.g., `logrotate` has both `logrotate.timer` and a cron entry in some distros; `mlocate`/`plocate` updatedb, `man-db`, etc.). Packaging best practice now is timer-only on systemd systems, with cron files conditional on non-systemd. So this scenario mirrors a real packaging/deployment bug: deploy script writes to /etc/cron.d AND installs the timer unit + enable. ## Root cause considerations Why did both get registered? - Deploy script (idempotency bug): it appends to /etc/cron.d unconditionally AND `systemctl enable --now myjob.timer` - Config management drift: old cron entry from Chef/Ansible not purged when timer introduced - Package upgrade shipped a timer while local cron file remained ## Resolution without breaking either mechanism "Without breaking either mechanism" — I read this as: the fix shouldn't damage cron or systemd timer subsystems themselves (can't just disable cron entirely or mask all timers), and both mechanisms should remain functional for other jobs. We need to make the *job* fire exactly once. Options: **Option A: Remove one registration (the eventual fix), keep both mechanisms healthy.** - Keep systemd timer, remove cron entry. Reasons below. **Option B: Mutual-exclusion lock (interim mitigation).** - Add `flock` around the job body: `flock -n /run/lock/myjob.lock /usr/local/bin/myjob.sh` in both cron entry and the service ExecStart. This prevents concurrent overlap but NOT back-to-back double-fires (if cron fires at 02:00 and timer at 02:00:30 after cron finished, both run sequentially). So flock alone is insufficient for "same job registered twice" unless the schedules are identical and executions overlap. If the job is long-running and schedules are aligned, flock would prevent concurrency but the second invocation still runs after the first — executing the work twice. So flock is only a partial mitigation. Important nuance to mention. - A stronger idempotency guard: job checks "already ran in window X" (e.g., lock file with timestamp, or state marker) — but that's re-architecting the job. - systemd-level: the timer's service can be made to not start if already running (that's default behavior for a single unit: `systemctl start myjob.service` while running does nothing by default, since Type=oneshot without RemainAfterExit... actually for Type=oneshot, concurrent start is blocked while active). But cron doesn't go through the unit unless ExecStart is the unit. You could make cron call `systemctl start myjob.service` instead of the binary directly! That unifies the execution path: cron becomes a trigger for the same unit; systemd's job semantics then dedupe concurrent starts. That's a clever "don't break either mechanism" fix: cron entry becomes `systemctl start myjob.service --no-block`? Hmm, but root cron can do `systemctl start myjob.service`. Then whether cron or timer fires first, the second start call is a no-op while the unit is activating/active. But if they fire at slightly different times and the job is quick, the first could complete before the second start → still runs twice. Type=oneshot units: while active (running), further starts are no-ops. After completion (inactive), a new start runs again. So dedupe only during overlap. Same limitation as flock. So true resolution is: remove one registration. The interim mitigations only prevent overlap, not duplicate sequential runs. **Option C: Idempotent job design** — job itself checks state (e.g., "did I already process today's batch?"). Good defense-in-depth but doesn't address the config bug. ## Which to remove and why: remove cron, keep systemd timer Reasons to prefer the systemd timer: 1. **Single source of truth & observability**: `systemctl list-timers`, `journalctl -u myjob.service` — logs, status, and scheduling are all in one place. Cron logs are scattered (syslog, /var/log/cron, mail spools). 2. **Dependencies and ordering**: timer units can have `Requires=`, `After=`, `Wants=network-online.target`, run after mount/network availability. Cron has no dependency concept. 3. **Resource controls**: `Nice=`, `CPUQuota=`, `IOSchedulingClass=`, `MemoryMax=`, cgroup accounting — cron children are just processes (though on modern systems with cronie, they're in a session, not a managed cgroup with per-job controls). 4. **Missing-run catch-up semantics**: `Persistent=true` with `OnCalendar` runs the missed job on next boot if the machine was down. Cron silently skips (unless anacron for the daily dirs). 5. **Randomized delay / accuracy**: `RandomizedDelaySec`, `AccuracySec` to avoid thundering herd; cron can't (well, you can sleep in the script, hacky). 6. **Environment & security**: unit file gives clean, explicit `Environment=`, `User=`, `WorkingDirectory=`, `UMask=`, `ProtectSystem=`, `PrivateTmp=`, sandboxing hardening. Cron env is brittle (PATH surprises are a classic). 7. **No mail-spool failure mode**: cron mails stdout/stderr; on systems without a configured MTA, output vanishes or spools. journald captures service output natively. 8. **State inspection**: `systemctl status` shows last/next trigger, result (failed/success), `systemctl list-timers` shows next elapse. Cron offers nothing comparable. 9. **Timer is already enabled and is the "newer" mechanism introduced by this deploy**: presumably the deploy intends the timer as the forward path; cron entry is legacy drift. Removing cron aligns with the deployment's intent. Also avoids the "two configs drift apart" trap (someone edits the timer, forgets the cron file, silently changes behavior... actually removing one fixes drift). 10. **Modern distro direction**: packaging guidance (Debian/Fedora) is to prefer timers on systemd systems; cron.d fragments are legacy. Counterpoints to acknowledge (why one might keep cron): exotic schedules cron can't express? Actually systemd can express cron schedules plus more. LiveCD/chroot without PID1? Not relevant post-deploy on a running server. Very fine-second scheduling? Neither cron (minute granularity) nor typical timer needs... timers support OnCalendar with sub-second actually. Cron's advantage is familiarity and no dependency on PID1 — but on a systemd server, PID1 is there. Edge case worth noting: if other jobs on the box are cron-managed, don't disable the cron daemon — just remove the specific line/file. "Without breaking either mechanism" = keep crond running for other jobs, keep timer units for other jobs; only deregister this job from one scheduler. Also check for hidden duplications after cleanup: - `grep -rn 'myjob' /etc/cron* /var/spool/cron` post-removal - `systemctl list-timers` confirm single timer - Verify next run: `systemctl list-timers myjob.timer` - Watch first scheduled run in `journalctl -f -u myjob.service` and cron logs to confirm exactly one execution. - Also confirm no other timers trigger the same service (`systemctl list-timers --all` and grep for the service: `systemctl list-dependencies --reverse myjob.service`? Actually `systemctl show myjob.service -p TriggersBy`? Hmm — `systemctl list-timers` shows the unit each timer activates. You can also check `grep -l myjob.service /etc/systemd/system/*.timer /usr/lib/systemd/system/*.timer`.) - Also watch for package reinstall re-adding cron file (dpkg/rpm owned file?) — check if /etc/cron.d entry belongs to a package: `dpkg -S /etc/cron.d/myjob` or `rpm -qf`. If it's from the package, removing the file means it returns on upgrade; instead disable via config management or use `dpkg-divert`/`systemctl mask`... If it's a config-management artifact, fix the recipe to purge it. Also mention `systemctl disable --now` vs `mask`: we're NOT masking the timer (we keep it); we remove cron entry: delete /etc/cron.d/myjob line or `crontab -e` for user crontab. For spool crontabs: `crontab -r` removes all — prefer editing. Reload not needed for cron.d (cron rescans), but note some crons need nothing; systemd daemon-reload after unit changes. Idempotency guard as defense-in-depth even after the fix: `flock -n` in the ExecStart wrapper prevents overlapping runs (useful if runtime can exceed interval). But emphasize: lock prevents concurrency, not duplicate sequential execution; the real fix is single registration. Also: Persistent=true — if both were firing and one was persistent... n/a after cleanup. Timezone check: `timedatectl`, `CRON_TZ`, `TZ=` in unit. If the "same schedule" was actually offset by TZ, you'd see double-fires at different times; journal timestamps reveal this. Worth including in verification since it affects what "double-fire" looks like. One more verification subtlety: `Persistent=` + machine rebooting could cause catch-up runs; also `OnBootSec`/`OnUnitActiveSec` timers vs OnCalendar — monotonic timers fire relative to boot/activation, so a timer + cron both firing "daily" could drift. Also potential cause: `systemd-cron` package (cron implemented via transient timers) — then /etc/cron.d entries are already timers; a *separate* native timer for same job = duplication. `systemctl list-timers` would show cron-*.timer units. If that's present, the fix differs: remove the native timer OR the cron file; same logic. Maybe worth a brief mention as a diagnostic wrinkle: check whether cron entries are being translated (packaging `systemd-cron`) — but that might be too niche. Brief parenthetical. ## Format considerations The user's message is terse, imperative, ops-flavored ("Same, plus: ... Verify and resolve... state which you would remove and why"). This reads like they want a direct, practical answer — likely a checklist-style response with commands. Given the phrasing, they're probably an SRE/devops person or working through a scenario set. The ideal response: - Verification section with concrete commands (block them or inline code) - Decision on resolution with the key insight (flock/lock mitigations are insufficient for back-to-back dupes; removal is the real fix) - Clear verdict: remove cron entry, keep timer, with reasons (bulleted, concise) - Post-fix verification steps - Watch-outs (package-owned cron file, config management drift, don't disable crond globally, TZ mismatch) Length: medium. Use headers and code blocks for commands. Keep prose tight. This is a technical audience; markdown is appropriate. Aim for something scannable — maybe 350-550 words plus code blocks. Avoid over-long essay. I'll structure: 1. **Verify** — enumerate both schedulers, confirm same target, confirm both schedules resolve to the same time, evidence from logs. 2. **Why locks don't fix it** (brief but crucial) — flock prevents overlap only; sequential double-run still happens. So removal is required. 3. **Remove cron, keep timer** — reasons. 4. **Safe execution order / steps**. 5. **Post-change verification**. 6. **Watch-outs** (package-owned file, config mgmt, don't touch crond itself, systemd-cron nuance, TZ). Let me draft the commands: Verify cron side: ``` crontab -l # root crontab -l -u grep -rn 'myjob' /etc/crontab /etc/cron.d/ /etc/cron.{hourly,daily,weekly,monthly}/ /var/spool/cron/ 2>/dev/null ``` Verify timer side: ``` systemctl list-timers --all | grep -i myjob systemctl cat myjob.timer myjob.service systemd-analyze calendar '' # confirm it resolves to same wall-clock time as the cron line ``` Confirm same target: - Compare command in cron line vs `ExecStart=` (resolve wrappers/symlinks: `readlink -f`) Evidence of double-fire: ``` journalctl -u myjob.service --since -7d --output short-iso | head grep CRON /var/log/syslog | grep myjob # or journalctl -t cron / -u cron # look for two runs per day, one logged by CROND, one by systemd ``` Check what triggers the service: ``` grep -rl 'myjob.service' /etc/systemd/system /usr/lib/systemd/system | grep timer systemctl list-timers --all # NEXT column ``` Timezone sanity: ``` timedatectl; grep -i tz /etc/cron.d/myjob; systemctl show myjob.timer -p Environment ``` Interim mitigation note: ``` # cron 0 2 * * * flock -n /run/lock/myjob.lock /usr/local/bin/myjob.sh # unit ExecStart=/usr/bin/flock -n /run/lock/myjob.lock /usr/local/bin/myjob.sh ``` ...prevents concurrent overlap; does NOT prevent the second firing after the first finishes. So it's a band-aid while you do the real fix. Alternatively cron triggers the unit: `0 2 * * * systemctl start myjob.service` — same dedupe limitation, but consolidates logging. Might mention as optional. Removal steps: ``` # remove cron registration (choose the right location) rm /etc/cron.d/myjob # or edit the line out crontab -e -u # if in a user spool # do NOT systemctl stop/disable cron itself # keep timer as the single registration systemctl enable --now myjob.timer systemctl daemon-reload # only if unit files changed ``` If cron file is package-owned: `dpkg -S /etc/cron.d/myjob` / `rpm -qf /etc/cron.d/myjob` — then fix at the source (config mgmt / package) or it resurrects on upgrade. Post-fix verification: ``` grep -rn 'myjob' /etc/cron* /var/spool/cron 2>/dev/null # expect empty systemctl list-timers myjob.timer # one timer, NEXT set # on next scheduled run: journalctl -f -u myjob.service grep CRON /var/log/syslog | grep myjob # expect silence from cron ``` Why remove cron / keep timer (concise bullets): - Observability: journald + `systemctl status`/`list-timers` vs scattered syslog/mail - Dependencies: After=/Requires=, network-online - Catch-up: Persistent=true reruns missed windows after downtime; cron silently skips - Hardening/limits: User=, Environment=, Nice=, CPUQuota=, sandboxing directives - Precision controls: AccuracySec, RandomizedDelaySec - No MTA dependency for capturing output - It's the registration the deploy intended (cron line is drift/legacy); single source of truth prevents future config drift - Matches distro packaging direction (timers preferred on systemd hosts) Keep cron itself running for remaining jobs; both mechanisms remain intact for everything else — that satisfies "without breaking either mechanism." One more nuance: if the job must run on hosts without systemd (mixed fleet), then the *package* should carry both but install conditionally — outside scope but worth one line? Maybe skip or one-liner. Also `systemd-cron` check: `systemctl list-timers | grep cron` — if distro translates cron→timers, the cron.d file isn't a second mechanism per se; but a native timer + cron still double-fires. One parenthetical line max. Actually I'll include it briefly in verification since it changes what you remove (you'd still remove the cron fragment, but the mechanism observation is different). Hmm, keep it short or cut. I'll include one short line — it's a genuinely common trap on some distros (e.g., Artix, or Debian with systemd-cron installed). Eh — risk of noise. I'll put it as a one-line parenthetical in watch-outs. Also: after removing cron entry, no restart needed — cron re-reads /etc/cron.d on change (most implementations stat it each minute). Note it so they don't unnecessarily bounce crond (which "breaking either mechanism" might be sensitive to). Format decision: headers with `**bold**` or `##`? Given multiple sections and command blocks, small `##` headers or bold lead-ins are fine. I'll use bold section labels or ### headers, command blocks, tight bullets. Target maybe 450-600 words total prose + commands. The user's style is terse; they want the answer, not an essay. I'll keep reasoning compact, verdict prominent. Let me write it. ## Verify the double-fire **1. Enumerate both schedulers:** ```bash # cron side crontab -l; crontab -l -u grep -rn 'myjob' /etc/crontab /etc/cron.d/ /etc/cron.{hourly,daily,weekly,monthly}/ /var/spool/cron/ 2>/dev/null # systemd side systemctl list-timers --all | grep -i myjob systemctl cat myjob.timer myjob.service ``` **2. Confirm they hit the same target** — compare the cron command vs `ExecStart=` (resolve wrappers/symlinks with `readlink -f`; a wrapper that looks different but calls the same binary still counts). **3. Confirm schedules actually collide:** ```bash systemd-analyze calendar '' # resolve timer spec to wall-clock timedatectl # TZ mismatch can offset the two fires ``` **4. Get empirical evidence** — two runs per interval, one logged by cron, one by systemd: ```bash journalctl -u myjob.service --since -7d --output short-iso journalctl -t cron --since -7d | grep myjob # or grep CRON /var/log/syslog ``` ## Why locks alone don't fix it Adding `flock -n /run/lock/myjob.lock` to both call sites prevents **concurrent** overlap only. If cron fires at 02:00:00 and the timer at 02:00:30 after the first run finished, the job still executes twice sequentially. Same limitation if cron calls `systemctl start myjob.service` instead of the binary — oneshot dedupe only applies while the unit is active. A lock is a reasonable **interim** mitigation, but the actual fix is deregistering one instance. ## Verdict: remove the cron entry, keep the timer Keep crond running for everything else — you're deleting one registration, not breaking the mechanism. Reasons the timer wins: - **Observability**: `systemctl status`, `list-timers`, journald capture. Cron output is scattered across syslog/mail spools (or lost with no MTA). - **Catch-up semantics**: `Persistent=true` reruns missed windows after downtime; cron silently skips. - **Dependencies & hardening**: `After=network-online.target`, `Requires=`, `User=`, `Environment=`, `Nice=`, `CPUQuota=`, sandboxing directives — none available via cron. - **Precision controls**: `AccuracySec`, `RandomizedDelaySec`. - **Intent**: the timer is what the deploy installed; the cron fragment is drift. Keeping the timer gives one source of truth and stops the two configs diverging silently. - Matches distro packaging direction (timers preferred on systemd hosts). ```bash # remove only the registration rm /etc/cron.d/myjob # or edit the line out / crontab -e -u systemctl daemon-reload # only if unit files changed systemctl enable --now myjob.timer # ensure it's the surviving registration ``` ## Post-fix verification ```bash grep -rn 'myjob' /etc/cron* /var/spool/cron 2>/dev/null # expect empty systemctl list-timers myjob.timer # exactly one, NEXT set # watch the first scheduled run journalctl -f -u myjob.service journalctl -t cron | grep myjob # expect silence ``` ## Watch-outs - **Resurrection**: if the cron file is package-owned (`dpkg -S /etc/cron.d/myjob` / `rpm -qf`), it returns on upgrade — fix at the source (package or config-management recipe), not just on disk. - **Check for `systemd-cron`**: if installed, `list-timers` shows `cron-*.timer` units generated *from* the cron fragment — same cleanup path, but be aware you're not looking at two independent files. - **Defense in depth**: still wrap `ExecStart` in `flock -n` — guards against long runs overlapping the *next* legitimate fire. - Don't `mask` the timer and don't touch crond — other jobs on the box depend on both mechanisms staying healthy.

SCB SCB-deploy_verify-T2 MUSE