We need answer user. Need infer context. "Same, plus..." likely previous conversation? We only have this. Need answer likely about after deploy, cron and systemd timer both register same job. Verify and resolve double-fire risk without breaking either mechanism; state which remove and why. Need craft response with commands to verify and resolution. Maybe "Same, plus" means likely from sysadmin/devops question. Need not know previous. Need handle generically. Need parse: "Same, plus: after deploy, cron and a systemd timer BOTH register the same job. Verify and resolve double-fire risk without breaking either mechanism; state which you would remove and why." They want probably operational procedure. Need answer: verify by listing cron entries and systemd timers; inspect OnCalendar, calendar next runs, overlap lock. Need resolve without breaking either mechanism: likely use flock to guard job so only one instance runs; but if both schedule same time, both invoke command, one acquires lock and other exits. This prevents double-fire without disabling either mechanism. Then state which mechanism remove and why: remove cron, keep systemd timer because better logging, dependencies, jitter, Persistent, RandomizedDelaySec, resource controls, central unit file; unless existing environment relies on cron (e.g. non-systemd systems), in which case remove systemd. Need verify and resolve. The prompt "without breaking either mechanism" maybe means keep both initially with lock? But "state which you would remove and why" means ultimately remove one to avoid duplicate. Could be: verify, mitigate via lock, then choose remove cron. Need answer maybe mention if they both "register" same job after deploy, deploy bug in installation logic (should be idempotent and gate on init system). We need give commands. Need decide assumptions: Linux systemd system cron? There may be user cron vs system cron. Need include both. Let's think of robust answer: 1. Capture intended job definition: - command, schedule, environment, user. 2. Verify cron: - `crontab -l -u appuser` or root: `crontab -l` - `/etc/crontab`, `/etc/cron.d/*`, `/etc/cron.hourly|daily|...`, spool: `grep -R --line-number -E '/usr/local/bin/job|job-name' /etc/cron* /var/spool/cron/crontabs /var/spool/cron 2>/dev/null` - `run-parts --test /etc/cron.daily` - For Debian: `grep -R ... /etc/cron.d /etc/crontab /var/spool/cron/crontabs` - `systemctl list-timers --all | grep job` 3. Verify systemd: - `systemctl cat job.timer` - `systemctl list-timers --all | grep -F job.timer` - `systemctl show job.timer -p OnCalendar -p OnUnitActiveSec -p Persistent -p RandomizedDelaySec -p NextElapseUSecRealtime` - `systemctl status job.timer` - `journalctl -u job.timer -u job.service --since ...` - `systemd-analyze calendar '...'` to confirm same schedule and fire time. 4. Compare next execution times: - `systemctl list-timers job.timer` - cron doesn't show next run easily; can parse or change schedule to near future for canary? Better use logs/audit. 5. Confirm actual double-fire: - Add unique id/start/end logging to script. - Use `journalctl` and cron logs `/var/log/cron`, `/var/log/syslog`, `grep CRON`. - Or `auditctl`? For production safer maybe not. - Check process logs; if job writes "start" and "done" with timestamp, see two starts at same minute. - Check deploy logs, package logs: e.g. install script wrote cron file and systemd units. `dpkg -S`, `rpm -qf`. 6. Resolve without breaking either mechanism: - Immediate mitigation: add a single-instance advisory/mandatory lock with `flock` to the command used by both cron and systemd. This does not disable either, ensures if both trigger simultaneously, only one executes; the other exits or waits. - Example wrapper `/usr/local/bin/job-run`: ``` #!/bin/sh set -eu exec 9>/run/lock/job-name.lock if ! flock -n 9; then logger -t job-name "another instance is running; skipping" exit 0 fi logger -t job-name "starting" # real job... logger -t job-name "finished" ``` - Ensure `/run/lock` writable by the running user? For system jobs root; for app user create `/run/lock/`? Better use `/var/lock` or `/run/lock` with ownership. Systemd service can use `RuntimeDirectory=job`? But cron doesn't create. Could use `/var/lib/job/lock` with proper ownership. Example: ``` LOCK=/var/lib/job/lock mkdir -p /var/lib/job touch "$LOCK" ... ``` But locking needs open fd; `flock` command. Could modify cron line: `* * * * * appuser /usr/bin/flock -n /var/lock/job.lock /usr/local/bin/job.sh` systemd service: `ExecStart=/usr/bin/flock -n /var/lock/job.lock /usr/local/bin/job.sh` Then both invoke same lock path. But with systemd, better avoid wrapper `flock` and use: ``` [Service] ExecStart=/usr/bin/flock -n /run/job.lock /usr/local/bin/job.sh ``` But if systemd service has `Type=forking`? Need handle. Better: `systemd-run`? No. Systemd service can use `ExecStart=/usr/local/bin/job-run` where wrapper has flock. Good. Could also use systemd `RefuseManualStart`? No. Need avoid "breaking either mechanism" means if we remove cron, systemd still works; if remove timer, cron still works? Wait phrase "without breaking either mechanism" maybe means they want to verify and resolve in a way that doesn't disrupt cron/systemd overall, not job? It says "without breaking either mechanism; state which you would remove and why." Contradictory? "both register same job. Verify and resolve double-fire risk without breaking either mechanism; state which you would remove and why." Maybe "without breaking either mechanism" could mean ensure deploy/removal doesn't corrupt cron or systemd. You remove duplicate from one mechanism but keep mechanism operational. "state which you would remove" implies choose one job registration to remove. Could mention lock as immediate mitigation then remove one scheduler after testing. Need include commands to remove duplicate safely: - If remove cron: - Identify exact source file. If in `/etc/cron.d/job`: remove via deploy manifest, not manual; `rm /etc/cron.d/job` or `sed -i '/pattern/d'`. Cron automatically notices changes in /etc/cron.d after a minute? Need reload? Cron daemon scans files periodically; for some distros restart/reload? Actually cron wakes every minute and rechecks crontab directories? `cron` reloads changed files. `/etc/cron.d` does not require explicit reload; but safe to `systemctl reload cron`? There is no reload maybe `systemctl restart cron` not recommended. For `crontab -u appuser -r` removes entire user's crontab; dangerous if multiple jobs. Better edit with `crontab -l | grep -v` and install only if no other lines. For file in `/etc/cron.d`, remove the file owned by package/deploy, but if shared file remove only the job line. - After removal, verify: `grep -R ...` no cron; `systemctl list-timers job.timer`; `systemctl daemon-reload` after unit changes? If already exists, no. If modifying service to lock, `systemctl daemon-reload`; `systemctl restart`? Timers not restart. Test `systemctl start job.service`. - Need ensure systemd service enabled? Timer enabled: `systemctl enable --now job.timer`. - If remove systemd: - `systemctl disable --now job.timer`; keep service file or remove unit. But question asks state which remove: remove cron. Keep systemd timer. - If systemd not present or compatibility required, remove systemd timer. Need mention "Persistent=true" can catch up missed runs; cron doesn't. If both have Persistent, risk after downtime: timer runs at boot and cron may run according to schedule? Need evaluate. If job should not catch up, set `Persistent=false`; if intended, lock still prevents duplicate catchup if same time, but if cron runs and systemd catchup at boot not same scheduled minute, maybe double execution separated. Need align semantics. Good point. Need maybe mention in systemd timer use `AccuracySec=1us`? Actually default AccuracySec=1min, which coalesces to minute; cron minute granularity. Could coincide. But double-fire risk if same schedule at same minute. Need use `RandomizedDelaySec` only in one? It can reduce collision, not guarantee. Lock is guarantee. `AccuracySec=1us` can make predictable. Need maybe mention "systemd timer has no $PATH and shell env unlike cron"; but service wrapper ensures same command. Might need mention `OnCalendar` equivalence: Cron `30 3 * * *` equals `*-*-* 03:30:00`; timezone differences: cron often local time, systemd uses local by default? `OnCalendar` uses local timezone unless specified. But `OnCalendar` with timezone might differ. Need verify. Commands: `systemd-analyze calendar 'Tue *-*-* 03:30:00'` No command for cron; can use `cronexpr` maybe. But compare. Could answer with "run a canary command that just logs, no-op, at a controlled near-future time to prove double registration." Example: - Replace job command temporarily with logger? But not in prod maybe. We can use wrapper canary in both definitions? Hmm. Need answer likely should be direct: ``` # Systemd timers systemctl list-timers --all | grep -F job systemctl cat job.timer # cron grep -R -n -- '/usr/local/bin/job' /etc/crontab /etc/cron.d /var/spool/cron 2>/dev/null crontab -l | grep job ``` Then: ``` systemd-analyze calendar "$(systemctl show job.timer --property=OnCalendar --value)" ``` Actually `systemctl show` might return multiple values separated by spaces? property maybe `OnCalendar` displays? For timers, property `OnCalendar`? Check: `systemctl show apt-daily.timer -p OnCalendar` maybe value `*-*-* 6,18:00`. Good. Need mention check service vs script lock: ``` systemctl cat job.service ``` Look at `ExecStart`, `User`, `WorkingDirectory`, `EnvironmentFile`. Need compare cron line to service `ExecStart` and user. Both same job if command and user match. If schedule same (not necessarily exact). Could use `systemctl list-timers --all --no-pager` and `systemd-analyze calendar`. For cron in `/etc/cron.d`, line includes user field: `30 3 * * * appuser /usr/local/bin/job` For user crontab, no user field. Double-fire risk if same command and next fire time same. If schedules differ but job can overlap if long-running or catchup. Use lock. Need maybe include auditd: ``` grep -E 'CRON|systemd\[1\].*Starting job' /var/log/syslog journalctl -u cron -u job.service --since '24 hours ago' ``` Actually cron logs to syslog, not journal? On Ubuntu, can `journalctl -t CRON -t systemd -u job.service`. Need maybe "strace" not needed. Need maybe include not to remove `crond`/systemd entirely, remove duplicate registration only. "not break either mechanism" maybe maybe they mean do not break service mechanism if remove cron? But final answer can say "Don't disable cron daemon; remove only the duplicate cron entry." Need think if "same plus" might be in context of "post-deploy verification checklist"? Need answer perhaps in a style: "Add this to runbook." They may want "Verify and resolve double-fire risk... state which remove." We can produce runbook. Potential pitfalls: - `flock -n` returns exit code 1 if lock not acquired. For systemd service, if ExecStart uses `flock -n`, service result may be "failure" when skipped. Better use wrapper that returns 0 on lock contention. Or use `flock -n -E 0`? Does flock have `-E, --conflict-exit-code ` option. `flock -n -E 0 lock command` returns 0 when lock conflict? Let's verify: util-linux flock options: `-E, --conflict-exit-code ` exit code used when conflict and `--no-flock`? Usually if conflict with -n, exit 1. `-E 0` sets exit code to 0. Yes. So can do `flock -n -E 0 /var/lock/job.lock /usr/local/bin/job`. But for cron, exit 0 prevents mail. Good. - However `-E 0` may not be available on older util-linux. Use wrapper robust. - Lock path on Debian/Ubuntu `/run/lock`, `/var/lock` symlink to `/run/lock`. Need parent exists. If service `User=appuser`, appuser may not have permission to create in `/run/lock`? Usually /run/lock mode 1777? Let's check; `/run/lock` often drwxrwxr-t root:root or 1777. Fine. But lock file created by appuser may persist. Better create lock file at deploy with ownership: ``` install -d -o appuser -g appuser /var/lib/job touch /var/lib/job/lock chown appuser:appuser /var/lib/job/lock ``` But for immediate wrapper can use `/var/lock/job-name.lock`; parent often writable. Or root service can write. - Need lock only prevents simultaneous runs, not duplicate runs at different times. If cron at 03:30, systemd timer has catchup at boot 03:35? Both won't overlap but will run twice. "double-fire" can mean duplicate invocation even if not concurrent. To truly avoid duplicate schedule, remove one mechanism. Lock is immediate safety for concurrent double-fire. For any duplicate scheduling at different times, only one scheduler should own job. The prompt asks "state which remove". So final: verify, lock as immediate/defense-in-depth, then remove cron registration and keep systemd timer. - For resolving "without breaking either mechanism": use a shared lock in both commands while both registrations exist. Then remove duplicate from one mechanism; that doesn't break the overall mechanism. To not break job during transition: deploy lock first, test systemd timer, then remove cron. Or disable cron line with comment? Let's outline safe sequence: 1. Add lock to real job/wrapper. Deploy wrapper. 2. Change both cron and systemd to invoke wrapper (if not already). 3. Test both schedulers manually: - `systemctl start job.service` - simulate cron by running the same command as same user/environment, or wait for scheduled run; but manual invocation not proving schedule. Check logs show one start, one skip. 4. Remove cron registration from package/deploy; not cron daemon. 5. `systemctl enable --now job.timer` if not already; ensure timer active. Verify no cron entries. 6. Test after deploy: timer active and job fires exactly once; `systemctl list-timers job.timer`; logs. 7. Keep lock in service as idempotence/overlap guard. Need maybe include "cron and systemd service often run with different users/environments; use same wrapper and User= in systemd; verify user with `systemctl show job.service -p User -p Group -p ExecStart` and cron user field or crontab owner." Need maybe mention if job is in anacron? Cron daily may use anacron, systemd timers apt-daily? Both same job? Need check `/etc/cron.daily`, `/etc/cron.hourly`, `/etc/anacrontab`, systemd calendar. If `/etc/cron.daily` script and timer both maybe. Use: ``` run-parts --test /etc/cron.daily ``` But generic. Need include maybe package idempotency: Deploy should choose exactly one scheduler: - in Ansible: ``` - name: Remove legacy cron ansible.builtin.file: path: /etc/cron.d/job state: absent - name: Install/enable timer systemd: name: job.timer enabled: true state: started daemon_reload: true ``` But maybe not needed. But could mention deploy script should not install both unconditionally. Need maybe answer "which would remove and why" maybe choose "remove cron, keep systemd timer" or maybe "remove systemd timer, keep cron"? Need state clearly. Which is better? Usually systemd timer is better for: - centralized visibility via systemctl/journalctl. - service sandboxing/resource limits. - dependencies (After=, RequiresMountsFor=, NetworkOnline) - better randomized delay, persistent catch-up. - avoids email side effects and environment quirks. - easier package management. Cron advantages: - ubiquitous, minimal. If app is portable to non-systemd systems or existing cron dependencies, then keep cron instead. But in modern Linux with systemd already used, keep systemd. The prompt likely expects that. Need maybe mention if system doesn't run systemd (containers, minimal images) then keep cron because timer can't work. But if both exist and systemd is PID 1, remove cron. Need maybe mention "remove cron registration, not the cron daemon or package" because cron might manage other jobs. "Without breaking either mechanism" maybe if remove cron job, cron mechanism still okay. Need emphasize. Need maybe include exact commands for current logs and verification. Let's design a thorough but concise runbook. Potential command examples: ``` JOB_PATTERN='myjob' # command or marker CMD_PATTERN='/usr/local/bin/myjob' # Cron crontab -l | grep -n "$CMD_PATTERN" # current user for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u "$u" 2>/dev/null | grep -n "$CMD_PATTERN" && echo "user=$u"; done grep -RIn --exclude='*.dpkg*' -E "$CMD_PATTERN|myjob" \ /etc/crontab /etc/cron.d /etc/cron.hourly /etc/cron.daily \ /etc/cron.weekly /etc/cron.monthly /etc/anacrontab \ /var/spool/cron /var/spool/cron/crontabs 2>/dev/null ``` Maybe the for loop might be heavy but okay. Could simply root uses: ``` grep -RIn "$CMD_PATTERN" /var/spool/cron /var/spool/cron/crontabs /etc/cron* /etc/anacrontab 2>/dev/null ``` But user crontabs are in /var/spool/cron/crontabs on Debian, `/var/spool/cron` on RHEL maybe. Need mention distro paths vary. ``` # Systemd systemctl list-timers --all --no-pager | grep -Ei 'myjob|job\.timer' systemctl cat myjob.timer systemctl cat myjob.service systemctl show myjob.timer -p ActiveState -p Unit -p OnCalendar -p OnBootSec -p OnUnitActiveSec -p Persistent -p RandomizedDelaySec -p NextElapseUSecRealtime systemd-analyze calendar "$(systemctl show myjob.timer -p OnCalendar --value)" ``` But if timer uses `OnBootSec` or `OnUnitActiveSec`, `OnCalendar` empty; need inspect cat. Use cat. Double-fire log: ``` journalctl -u myjob.service --since "24 hours ago" | grep -E 'Starting|Finished|Deactivated' grep -Ei 'myjob|CRON' /var/log/cron* /var/log/syslog* 2>/dev/null | tail -n 100 # systemd: journalctl -t CRON -u myjob.service --since "24 hours ago" ``` For cron logs: - RHEL: `/var/log/cron` - Debian: `/var/log/syslog` or `journalctl -t CRON` Command: ``` journalctl -t CRON --since "24 hours ago" | grep "$CMD_PATTERN" ``` Need show how to manually test timer and lock: ``` # Do this during maintenance or with no-op version first sudo systemctl start myjob.service sudo -u appuser /usr/local/bin/myjob-wrapper ``` Actually manual start while lock held could show skip: - Run a long test lock: ``` sudo -u appuser flock -n /var/lock/myjob.lock -E 0 -c 'sleep 30' & systemctl start myjob.service journalctl -u myjob.service -n 20 ``` But if service starts wrapper and lock held, logs skip exit 0. Good test. Wrapper script: ``` cat >/usr/local/bin/myjob-wrapper <<'EOF' #!/bin/sh set -eu LOCK=/var/lock/myjob.lock LOG_TAG=myjob exec 9>"$LOCK" if ! flock -n 9; then logger -t "$LOG_TAG" "lock busy; skipping duplicate invocation" exit 0 fi logger -t "$LOG_TAG" "starting" # Put real command here, or exec it. /usr/local/bin/myjob rc=$? logger -t "$LOG_TAG" "finished rc=$rc" exit $rc EOF chmod 0755 /usr/local/bin/myjob-wrapper chown root:root /usr/local/bin/myjob-wrapper touch /var/lock/myjob.lock chown appuser:appuser /var/lock/myjob.lock # if job runs as appuser; if root, root ``` But note shell `set -e` with `rc=$?` after command if command fails: if `/usr/local/bin/myjob` fails, script exits immediately before logging. Need not use `set -e` or handle: ``` set -u ... if /usr/local/bin/myjob; then rc=0; else rc=$?; fi ... exit "$rc" ``` Better: ``` #!/bin/sh set -u exec 9>/var/lock/myjob.lock if ! flock -n 9; then logger -t myjob "another instance running; skipping" exit 0 fi logger -t myjob "start" /usr/local/bin/myjob rc=$? logger -t myjob "finish rc=$rc" exit $rc ``` Without `set -e`, capture rc. Good. Or `exec` not capture. Need if real job spawns children and exits before children complete, lock releases while children still running; then duplicate could happen. Need ensure wrapper waits for all children or systemd service tracks main process. Cron command runs script synchronously; if job daemonizes and returns, lock not enough. Mention if job forks, run it in foreground or use `Type=forking` with PIDFile or systemd service; flock only guards while foreground wrapper runs. Maybe not necessary but good nuance. Alternative use `flock` in cron and systemd service: ``` ExecStart=/usr/bin/flock -n -E 0 /var/lock/myjob.lock /usr/local/bin/myjob ``` and cron: ``` * * * * * appuser /usr/bin/flock -n -E 0 /var/lock/myjob.lock /usr/local/bin/myjob ``` But if logger skip not available. Good if concise: ``` ExecStart=/usr/bin/flock --nonblock --conflict-exit-code 0 /run/lock/myjob.lock /usr/local/bin/myjob ``` Maybe use this. Systemd native single-instance per service doesn't prevent cron launching same command. But systemd has `RuntimeMaxSec` etc. Lock is shared. Need maybe mention `systemd` service can have `ConditionPathExists`? No. Could use ` systemd-run`? Not relevant. Potential answer structure: ``` Assume job name is myjob and command /usr/local/bin/myjob. Replace values. 1. Verify: ... 2. Mitigate: ... 3. Remove duplicate: ... 4. Which I would remove: ... ``` Need maybe include exact removal commands: For cron removal: - If in `/etc/cron.d/myjob`: remove file: ``` rm /etc/cron.d/myjob ``` - If in user crontab: ``` crontab -l -u appuser | grep -vF '/usr/local/bin/myjob' | crontab -u appuser - ``` Danger if same line appears? It removes all matching. Need preserve comments. Use carefully. Could `crontab -l -u appuser > backup`, `crontab -u appuser -r` only if no other entries. Avoid `-r`. - If in `/etc/crontab` or shared `/etc/cron.d/*`, edit with `sed -i` or config management: ``` sed -i '\|/usr/local/bin/myjob|d' /etc/crontab ``` But state use deploy manifest to avoid reintroduction. Check no duplicate: ``` grep -RIn '/usr/local/bin/myjob' /etc/cron* /var/spool/cron* 2>/dev/null ``` Note shell glob maybe `/etc/cron*` includes `/etc/cron.d` etc. Good. Enable timer: ``` systemctl daemon-reload systemctl enable --now myjob.timer systemctl start myjob.service # manual smoke test systemctl list-timers myjob.timer --no-pager journalctl -u myjob.service -f ``` If timer already active, `daemon-reload` after changing unit, `systemctl restart myjob.timer` maybe not necessary? Unit changes might apply; use `systemctl restart myjob.timer` to reload timer settings. But restart timer can reset next schedule? In systemd, timer units often "active"; after `daemon-reload`, changes apply; "systemctl restart" might trigger immediately? Usually restart timer doesn't start service? Timer remains active. Better: ``` systemctl daemon-reload systemctl restart myjob.timer ``` Safe generally. For lock wrapper changed service: ``` systemctl daemon-reload ``` No need to restart service except manual. Need if removing cron, no need to restart crond because cron rescans; but to be explicit: ``` # Do not restart crond for /etc/cron.d; if your distro requires: systemctl reload cron || true ``` Need be careful `systemctl reload cron` may fail. Could say "Don't disable cron; changes in cron files are picked up automatically; if your distro documents a reload, use it." Maybe no. Need mention systemd timer can mask cron? No. Need maybe mention `systemctl disable --now job.timer` if choosing remove systemd instead: ``` systemctl disable --now myjob.timer rm /etc/systemd/system/myjob.{timer,service} systemctl daemon-reload ``` But since we choose keep systemd, not include? Maybe include as contingency. Need think if "after deploy, cron and a systemd timer BOTH register the same job" could be due to package has both files; perhaps "after deploy" in Ansible; maybe "Verify and resolve double-fire risk without breaking either mechanism; state which you would remove and why" Maybe they expect a "post-deploy sanity check" with commands. We can answer "I would remove the cron entry and keep the systemd timer. Because systemd has explicit unit state, logs, retries? Actually systemd timers don't retry failed service by default; cron sends email; but systemd has dependencies, sandboxing, randomization, catchup." Need maybe mention "systemd timer is not inherently retried; you can add OnFailure, Restart? For timer service, `Restart=on-failure` is allowed; timers can `Persistent=true` for missed. If job failure alerts require `OnFailure`." But not necessary. Could mention "Cron is often disabled in minimal/containers; systemd timer already required by deploy; avoid cron environment surprises." Good. Need maybe mention "If cluster/multiple hosts, use distributed lock (DB lock, Redis lock) rather than local flock; local flock only prevents double-fire on the same host." Important: If cron and systemd timer on same host? The prompt likely same machine. But if multiple nodes share job, both mechanisms per host plus multiple hosts could double. Should mention "for single host; if job is cluster-wide use distributed lock or leader election." Need maybe include "time zone: systemd `OnCalendar` and cron may use different timezone. Verify with `timedatectl` and `Timezone` in `OnCalendar` expression; systemd uses local timezone unless prefix `UTC` or suffix." Let's explore: systemd calendar event syntax supports timezone at end: `Mon *-*-* 00:00:00 UTC`. Cron uses local system time, unless `CRON_TZ` in cron.d maybe. Use same timezone. If not, schedules may not collide but still duplicate at different times. The request specifically double-fire, maybe same time. But good to include. Need maybe include "Cron's @reboot and systemd OnBootSec could double at boot." If both job definitions include boot-time. Need inspect. - cron `@reboot appuser /usr/local/bin/job` in /etc/crontab or user crontab. - systemd timer `OnBootSec=5min` or `Persistent=true` may run after boot. Need check. Need maybe include "anacron might delay daily jobs; so actual firing not exactly schedule. Logs are ultimate verification." Good if job is daily. Potential answer maybe: ``` # Get exact systemd timer spec: systemctl cat app-job.timer systemctl list-timers --all --no-pager | grep app-job.timer systemd-analyze calendar 'Wed *-*-* 02:00:00' # Get exact cron: grep -RIn "app-job" /etc/cron* /var/spool/cron* /etc/anacrontab 2>/dev/null ``` If same schedule and command: ``` # Add lock to service: [Service] ExecStart=/usr/bin/flock --nonblock --conflict-exit-code 0 /run/lock/app-job.lock /usr/local/bin/app-job ``` and cron: ``` */5 * * * * appuser /usr/bin/flock --nonblock --conflict-exit-code 0 /run/lock/app-job.lock /usr/local/bin/app-job ``` But if using wrapper, show wrapper. Need choose one representation. Maybe show both? Avoid too long. But final desired oververbosity 5; moderate. We can include wrapper and one-liner alternative. Need think of "without breaking either mechanism" maybe if use `flock` in the systemd service's `ExecStart`, systemd's service may mark skipped as successful if `flock -E 0`. Good: ``` ExecStart=/usr/bin/flock -n -E 0 /run/lock/myjob.lock /usr/local/bin/myjob ``` The lock file must exist? `flock` can create it. If parent writable. For appuser, /run/lock sticky maybe. But for security, `/run/lock/myjob.lock` could be pre-created by package with ownership. If file absent and appuser has write permission, okay. If /run/lock is 1777, okay. But lock files in `/run` are lost on reboot, that's fine; created again. If both systemd and cron try to create simultaneously, flock handles? Yes open same path with O_CREAT, both open same inode if no races? O_CREAT atomic. Potential problem: If service uses `User=appuser`, `/run/lock` mode maybe 0775 root:lock, not appuser. Let's check distributions: Debian /run/lock maybe `drwxrwxrwt 1 root root`. Many appusers may create. Fine. If not, use `/var/lib/myjob/lock`. Need mention if jobs run as root or appuser; lock path must be writable by both identical identities. If cron runs as root and systemd `User=appuser`, they cannot share a lock if path permissions don't allow? They can if lock file is writable by both (root can, appuser if owner/perms). Use `User=` to match cron. Potential issue: cron and systemd may have different umask; lock file created by first might not be writable by second if permissions. Ensure `chmod 0666` or group. Better use root-owned sticky directory or `install` to create file with mode 0666: ``` install -d -m 0755 -o root -g root /run/lock touch /var/lock/myjob.lock chown appuser:appuser /var/lock/myjob.lock chmod 0600 ``` If both run same appuser, okay. If both root, okay. Better lock directory: ``` install -d -o appuser -g appuser -m 0755 /var/lib/myjob touch /var/lib/myjob/lock chown appuser:appuser /var/lib/myjob/lock chmod 0600 /var/lib/myjob/lock ``` But for root and appuser? Use group. Maybe simpler: ``` LOCK=/var/lock/myjob.lock install -m 0666 /dev/null "$LOCK" # if root and multiple users ``` But world-writable lock can be symlink attack? `/var/lock` sticky and root creates file maybe. Use package creates and owns. Need not dive too much. Need maybe include "Do not use `mkdir /run/lock/myjob` as `RuntimeDirectory` if cron needs it; systemd private runtime dirs may not persist for cron. Use a persistent path outside systemd-managed runtime if cron must access it." Potential answer final with "Assumptions": - `myjob.timer`/`myjob.service`, command `/usr/local/bin/myjob`, user `appuser`. Replace. Let's plan final answer sections: 1. Intro: "Treat it as duplicate scheduling and possible concurrent execution. First prove it, then add a lock, then remove one scheduler. I would keep systemd timer and remove cron entry." 2. "Verify both registrations" commands: - Cron: ``` CMD='/usr/local/bin/myjob' grep -RIn -- "$CMD" /etc/crontab /etc/cron.d /etc/cron.hourly /etc/cron.daily /etc/cron.weekly /etc/cron.monthly /etc/anacrontab /var/spool/cron /var/spool/cron/crontabs 2>/dev/null crontab -l -u appuser 2>/dev/null | grep -n "$CMD" ``` - Systemd: ``` systemctl list-timers --all --no-pager | grep -Ei 'myjob|myjob.timer' systemctl cat myjob.timer myjob.service systemctl show myjob.timer -p ActiveState -p LastTriggerUSec -p NextElapseUSecRealtime -p Persistent -p RandomizedDelaySec systemd-analyze calendar "$(systemctl show myjob.timer -p OnCalendar --value)" ``` - Logs: ``` journalctl -u myjob.service --since -24h journalctl -t CRON --since -24h | grep -F "$CMD" grep -RIn "$CMD" /var/log/cron* /var/log/syslog* 2>/dev/null | tail -100 ``` Need mention if both show same `ExecStart`/cron command and overlapping trigger time => double-fire risk. Maybe `systemctl show -p OnCalendar --value` maybe syntax? `systemctl show myjob.timer --property=OnCalendar --value` maybe okay. For multiple properties with `--value` output lines. Good. `systemd-analyze calendar` with empty string maybe issue. Fine. 3. "Immediate mitigation while both are installed" lock: - Use shared lock in both mechanisms: ``` # as root, pre-create lock with ownership appropriate to job user install -d -o appuser -g appuser -m 0750 /var/lib/myjob touch /var/lib/myjob/lock chown appuser:appuser /var/lib/myjob/lock ``` Actually if both run same user. If root, change accordingly. Use `/var/lock/myjob.lock` maybe easier. But `install -d /var/lib/myjob` weird package data. Let's use `/var/lock/myjob.lock`: ``` install -m 0666 /dev/null /var/lock/myjob.lock ``` World-writable maybe not ideal. Maybe: ``` touch /var/lib/myjob/lock chown appuser:appuser /var/lib/myjob/lock chmod 0600 /var/lib/myjob/lock ``` Need parent directory accessible. Use: ``` install -d -o appuser -g appgroup -m 0750 /var/lib/myjob install -o appuser -g appgroup -m 0600 /dev/null /var/lib/myjob/lock ``` But if root cron and appuser systemd, chgrp shared? Overkill. Maybe show: ``` LOCK=/var/lock/myjob.lock # if both run as same user, this is sufficient: touch "$LOCK" chown appuser:appuser "$LOCK" chmod 0600 "$LOCK" ``` But `touch` file if not exists; parent owned root. If appuser cannot create but root can precreate. Good. If both root, chown root. Then: ``` /usr/bin/flock -n -E 0 "$LOCK" /usr/local/bin/myjob ``` Set cron line: ``` 30 3 * * * appuser /usr/bin/flock -n -E 0 /var/lock/myjob.lock /usr/local/bin/myjob ``` Systemd service: ``` ExecStart=/usr/bin/flock -n -E 0 /var/lock/myjob.lock /usr/local/bin/myjob ``` Then `systemctl daemon-reload`. Test: ``` # hold lock: flock /var/lock/myjob.lock -c 'sleep 30' & systemctl start myjob.service journalctl -u myjob.service -n 20 ``` But if service uses same user, holding lock as root is okay. Then service should exit 0 due -E 0 and not run. Good. It logs maybe no custom skip; `flock -E 0` silent. Could add wrapper for logging. Maybe show wrapper for skip logging. To avoid too many examples, use one-liner and mention wrapper if you want logs. But verifying skip via `journalctl` service won't show command, just service started and exited success. Good enough maybe. But user asked verify; logs helpful. Maybe wrapper more robust. Let's include wrapper in final, perhaps as code block. It adds moderate length. We'll mention one-liner equivalent. The final answer can have both? Maybe too much. We can choose wrapper for clarity. Wrapper: ``` cat >/usr/local/bin/myjob-run <<'EOF' #!/bin/sh set -u lock=/var/lock/myjob.lock tag=myjob exec 9>"$lock" if ! flock -n 9; then logger -t "$tag" "lock busy; skipping duplicate invocation" exit 0 fi logger -t "$tag" "start" /usr/local/bin/myjob rc=$? logger -t "$tag" "finish rc=$rc" exit $rc EOF chmod 0755 /usr/local/bin/myjob-run ``` Cron: ``` 30 3 * * * appuser /usr/local/bin/myjob-run ``` Systemd: ``` ExecStart=/usr/local/bin/myjob-run ``` Pre-create lock file: ``` touch /var/lock/myjob.lock chown appuser:appuser /var/lock/myjob.lock chmod 0600 /var/lock/myjob.lock ``` Need if job runs root, chown root. If `logger` available. Test: ``` systemctl daemon-reload flock /var/lock/myjob.lock -c 'sleep 30' & systemctl start myjob.service journalctl -t myjob -n 20 ``` It should show "lock busy; skipping..." if working. If actual job takes 30 sec, wait. Use `flock` command directly: ``` flock /var/lock/myjob.lock -c 'sleep 30' & ``` Need run as root? okay. If lock file mode 0600 appuser, root can. If service runs appuser and lock held by root, appuser can open existing? It has owner appuser mode 0600, root created file owned appuser. appuser can open. Good. Holding lock doesn't change ownership. Fine. But `flock ... -c` syntax: `flock [options] ` or `flock [options] -c `. The latter uses shell? util-linux supports `-c, --command ` run command through shell. It may require `sh -c`. Fine. Need ensure wrapper uses `flock -n 9`; this syntax requires util-linux `flock` supports FD. Good. Important: If wrapper `exec 9>"$lock"` and file mode 0600 appuser, okay. If lock path in `/var/lock` disappears after reboot? `/var/lock` symlink to `/run/lock`; file disappears after reboot. If wrapper as appuser cannot create in `/var/lock`, startup after reboot may fail. So better precreate lock on boot (systemd tmpfiles) or use `/var/lib/myjob/lock` persistent. Since cron and systemd both need persistent path. Use `/var/lib/myjob/lock`: ``` install -d -o appuser -g appuser -m 0750 /var/lib/myjob install -o appuser -g appuser -m 0600 /dev/null /var/lib/myjob/lock ``` This persists across reboot and appuser can access. Better. But if app user has no group? Use appuser:appuser. For root job, root. Let's use `LOCK=/var/lib/myjob/lock`. Need parent. Good. But if job uses `RuntimeDirectory` with systemd, cron won't. Use `/var/lib`. Wrapper: ``` LOCK=/var/lib/myjob/lock ``` Precreate: ``` install -d -o appuser -g appuser -m 0750 /var/lib/myjob install -o appuser -g appuser -m 0600 /dev/null /var/lib/myjob/lock ``` If both run as same appuser. If not, align user. Need mention. Could use `/var/lock` if simpler. But `/var/lib` more robust. Potential problem: `install /dev/null` sets empty file with owner; if file exists, `install` overwrites/truncates every deploy? If job running during deploy, install may replace inode; lock semantics could fail (processes lock different inodes). Important! Do not replace lock file while service may be running. Use `touch` if not exists; or package creates initially. If using `install` every deploy while job running, bad. Need mention "create once; don't replace during deploys while job may be running." Use: ``` install -d ... [ -e "$LOCK" ] || install -o appuser -g appgroup -m 0600 /dev/null "$LOCK" ``` But if file exists, don't overwrite. In shell: ``` touch "$LOCK" && chown ... && chmod ... ``` Touch existing doesn't replace inode. But chown/chmod while running okay. Use root. If missing, create. Good: ``` install -d -o appuser -g appuser -m 0750 /var/lib/myjob touch /var/lib/myjob/lock chown appuser:appuser /var/lib/myjob/lock chmod 0600 /var/lib/myjob/lock ``` Touch as root creates file root-owned; then chown appuser. Good. If deploying and lock file already held, chown/chmod no issue. `touch` changes mtime but doesn't replace inode. Good. Could use `install` but mention not on each deploy if running. Let's avoid. Need maybe "if job runs as root, substitute root:root; don't make lock world-writable." 4. "Remove one registration": - Choose cron. "I would remove the cron entry, not the cron daemon." - Use config management and exact file: ``` rm -f /etc/cron.d/myjob ``` or edit user crontab: ``` crontab -l -u appuser | grep -vF '/usr/local/bin/myjob' | crontab -u appuser - ``` Caution: backup first and only remove this line; if only job, use `crontab -u appuser -r`; but dangerous. Use: ``` crontab -l -u appuser > /root/appuser.cron.bak crontab -l -u appuser | grep -vF '/usr/local/bin/myjob' | crontab -u appuser - ``` But if no output, `crontab` might install empty. Fine. But if command is in wrapper, grep wrapper. Use exact pattern. Need mention do not remove all cron. - Ensure systemd timer active: ``` systemctl daemon-reload systemctl enable --now myjob.timer systemctl restart myjob.timer systemctl list-timers myjob.timer --no-pager ``` - Verify no cron: ``` grep -RIn '/usr/local/bin/myjob\|myjob-run' /etc/crontab /etc/cron.d /var/spool/cron /var/spool/cron/crontabs 2>/dev/null ``` - Monitor: ``` journalctl -u myjob.service -f journalctl -t CRON -f ``` Wait one scheduled interval; expect one `start` and one `finish`, no duplicate. 5. "Which and why": - Keep systemd timer, remove cron because: - Single declarative unit, versioned/deployed with app. - Better observability: `systemctl list-timers`, `journalctl`, `systemctl status`. - Better control: dependencies, timeout, environment, user, sandboxing, CPU/memory limits, `Persistent`, `RandomizedDelaySec`, `AccuracySec`, `OnFailure`. - Avoids cron's implicit environment, mail-only failure behavior. - systemd is already in use because timer exists; cron remains for legacy jobs. - Exceptions: keep cron if platform lacks systemd, container, fleet requires cron, or operational ownership strongly tied to cron. In that case remove/disable timer and keep cron: ``` systemctl disable --now myjob.timer ``` and remove unit files, leaving cron. But default modern Linux: remove cron. Need perhaps mention "do not remove `cron.service`; it may be used by other jobs." Good. Need include "schedule equivalence and timezone" maybe in verify: - Compare cron expression with systemd `OnCalendar`; if systemd timer uses `OnCalendar`: ``` systemd-analyze calendar "*-*-* 03:30:00" ``` Cron `30 3 * * *` corresponds to local time. If systemd calendar has timezone suffix, adjust: ``` systemd-analyze calendar "30 3 * * *"? ``` No, systemd doesn't accept cron syntax. We can manually compare. Could mention "Check `timedatectl` and any `CRON_TZ`/`TZ` in cron file; ensure both use same time base." This is useful. Potential issue: systemd's `OnCalendar` property in `systemctl show` for timers maybe "OnCalendar=*-*-* 03:30:00" yes. If timer uses `OnUnitActiveSec=...`, no OnCalendar. Use `systemctl cat`. Need include "NextElapseUSecRealtime" shows timer's next time, and cron doesn't. Logs show actual. To verify actual double-fire after deploy: - Search logs for the same job start tag: ``` journalctl -t myjob --since "-24h" | grep start ``` With wrapper after deploy, if both fired at same time: ``` Jan ... appuser myjob[...]: start Jan ... appuser myjob[...]: lock busy; skipping duplicate invocation ``` This proves double registration. But without lock, two `start` lines prove. Add before? "If you have not added logging, temporarily run job with a canary that only logs instead of side effects." Maybe mention but not necessary. Need maybe "run once via both mechanisms manually after adding lock to prove lock path": - Cron manual: ``` sudo -u appuser /usr/local/bin/myjob-run ``` - Systemd: ``` sudo systemctl start myjob.service ``` But sequential not overlap. Need hold lock test. Need maybe "without breaking either mechanism" perhaps they might not want to choose until verify; but ask "state which you would remove." We can explicitly state "The lock is only a bridge/defense-in-depth; it does not eliminate duplicate non-overlapping executions. The permanent fix is to remove one registration." Let's think of actual double-fire mechanisms: - If cron and systemd timer both schedule at same minute, both invoke. If command is idempotent maybe okay, but still resource. Lock prevents if simultaneous. But if timer has `RandomizedDelaySec=5m`, systemd may run 5 minutes after cron. They both invoke same job twice not concurrent. The lock won't prevent. Removing one scheduler is required to eliminate duplicate. Need emphasize: "A lock prevents overlapping, not duplicate scheduled executions; if the same job is scheduled twice at different times, it will run twice." Good. This addresses "double-fire risk" more broadly. Could resolve without breaking either mechanism by "make the job registration conditional in deployment, choose scheduler based on environment; e.g., a wrapper service but only one timer/entry active." But "both register" maybe we need to remove duplicate. If job can be triggered by both at different times, no lock prevents double. So final must say permanent fix remove one. Maybe phrase: "Immediate safety: shared lock. Correct resolution: make deployment idempotent and remove one scheduler. I would remove the cron entry and keep the systemd timer." Need maybe include "If you need a transition period without removing either, run one mechanism with a flag disabled? You can disable one timer or cron line (not entire mechanism). For no double fire, one registration must not trigger. The lock only protects simultaneous runs." Good. Potential final response should not be too long, but includes commands. Desired oververbosity 5, but user asks technical, likely expects concrete. We'll include commands. Let's consider edge cases with "after deploy, cron and systemd timer BOTH register same job" maybe in a container/hosts: - If cron daemon and systemd timer both active in a Kubernetes pod? systemd may not work. But prompt says systemd timer registers, so systemd likely. - If job registered by package `foo-cron` and `foo-systemd` packages installed due to dependencies. Removing cron package may be better: ``` dpkg -l | grep -E 'foo-cron|foo-timer' apt-get remove foo-cron ``` or `dnf remove foo-cron`? Need mention if installed by package: remove the package/entry that installs cron, not just file, or deploy will re-add it. Use package manager: ``` dpkg -S /etc/cron.d/myjob rpm -qf /etc/cron.d/myjob ``` Then remove that package or adjust package dependencies. This is important because "after deploy" means manual removal will be undone. Include: ``` dpkg -S /etc/cron.d/myjob || rpm -qf /etc/cron.d/myjob ``` If package owns it, uninstall/disable package or fix packaging; don't `rm` only. Need maybe include "If this is deploy-managed, change deploy manifests so it is mutually exclusive; otherwise next deploy re-adds cron." Very important. We'll add. Need maybe include "Use a canary/no-op command at first" if high risk: - "Before pointing real job at both, test lock with `logger`/`sleep` to ensure no side effects." Good maybe. Let's draft final answer in polished style: ``` Assume job: myjob; service myjob.service; timer myjob.timer; command /usr/local/bin/myjob; runs as appuser. ``` Then code. Detailed command section: ``` CMD='/usr/local/bin/myjob' # systemd systemctl list-timers --all --no-pager | grep -E 'myjob|myjob\.timer' systemctl cat myjob.timer systemctl cat myjob.service systemctl show myjob.timer \ -p ActiveState -p Unit -p LastTriggerUSec -p NextElapseUSecRealtime \ -p OnCalendar -p OnBootSec -p OnUnitActiveSec -p Persistent -p RandomizedDelaySec systemd-analyze calendar "$(systemctl show myjob.timer -p OnCalendar --value)" # cron grep -RIn -- "$CMD" /etc/crontab /etc/cron.d /etc/cron.hourly /etc/cron.daily /etc/cron.weekly /etc/cron.monthly /etc/anacrontab /var/spool/cron /var/spool/cron/crontabs 2>/dev/null crontab -l -u appuser 2>/dev/null | grep -n -- "$CMD" # actual journalctl -u myjob.service --since -24h --no-pager journalctl -t CRON --since -24h --no-pager | grep -F "$CMD" ``` Maybe `journalctl --since -24h` syntax: It should be `--since "24 hours ago"` or `--since=-24h`. Does journalctl accept `-24h`? It accepts timestamp argument; `-24h` maybe not? Better use `--since "24 hours ago"`. `grep -RIn "$CMD" /var/spool/cron /var/spool/cron/crontabs` if directories don't exist, error suppressed. Good. Need if command in cron as wrapper, grep wrapper too: ``` grep -RIn -E "$CMD|myjob-run" ``` Maybe set `PAT='myjob|/usr/local/bin/myjob'`. In systemd show, `systemctl show` property `OnCalendar` may be absent if timer uses monotonic. Fine. For cron user-specific, `crontab -l -u appuser` only works as root. Good. Then "If output shows same command/user and schedules overlap, double-fire confirmed." Need "if package owns": ``` dpkg -S /etc/cron.d/myjob 2>/dev/null || rpm -qf /etc/cron.d/myjob ``` Maybe include in removal section. Lock section: ``` # Put both paths behind same wrapper. cat > /usr/local/bin/myjob-run <<'EOF' #!/bin/sh set -u tag=myjob lock=/var/lib/myjob/lock exec 9>"$lock" if ! flock -n 9; then logger -t "$tag" "lock busy; skipping duplicate invocation" exit 0 fi logger -t "$tag" "start" /usr/local/bin/myjob rc=$? logger -t "$tag" "finish rc=$rc" exit "$rc" EOF chmod 0755 /usr/local/bin/myjob-run # Create lock with correct ownership, do not replace it on every deploy install -d -o appuser -g appuser -m 0750 /var/lib/myjob touch /var/lib/myjob/lock chown appuser:appuser /var/lib/myjob/lock chmod 0600 /var/lib/myjob/lock ``` If job runs root, root. Then: ``` # cron line 30 3 * * * appuser /usr/local/bin/myjob-run # service ExecStart=/usr/local/bin/myjob-run ``` If they use one-liner: ``` flock -n -E 0 /var/lib/myjob/lock /usr/local/bin/myjob ``` Maybe not include to keep concise. But wrapper has skip logging and success. Test: ``` systemctl daemon-reload # hold lock for 30s and try service flock /var/lib/myjob/lock -c 'sleep 30' & systemctl start myjob.service journalctl -t myjob -n 20 ``` Expect "lock busy; skipping duplicate invocation", service success. But if `systemctl start` exits success because wrapper exit 0. Good. Potential issue: holding lock as root; service appuser opens existing file owned appuser and mode 0600. okay. Then "Permanent fix" remove cron: ``` # if deploy-managed, fix manifest/package so cron is not installed. rm -f /etc/cron.d/myjob # user crontab: crontab -l -u appuser > /root/appuser.crontab.bak crontab -l -u appuser | grep -vF '/usr/local/bin/myjob-run' | crontab -u appuser - ``` But if command in cron still original `/usr/local/bin/myjob`, grep wrapper maybe not match. In permanent fix after both point to wrapper, remove wrapper line. But if remove cron before pointing? Hmm. We'll say pattern: ``` crontab -l -u appuser | grep -vF '/usr/local/bin/myjob-run' ``` or `/usr/local/bin/myjob`. Need "use exact line/pattern found in verify." Good. Use `grep -vE 'myjob-run|/usr/local/bin/myjob'`. But removing via `crontab` pipeline can drop comments? It keeps all except matching. Good. For file in `/etc/crontab`, edit file: ``` sed -i '\|/usr/local/bin/myjob-run|d' /etc/crontab ``` But maybe not include because dangerous. Use config management. Then enable timer: ``` systemctl daemon-reload systemctl enable --now myjob.timer systemctl restart myjob.timer systemctl list-timers myjob.timer --no-pager ``` After removal: ``` grep -RIn -E 'myjob-run|/usr/local/bin/myjob' /etc/crontab /etc/cron.d /var/spool/cron /var/spool/cron/crontabs 2>/dev/null journalctl -u myjob.service --since "next scheduled window" ... ``` No actual "next scheduled window" command. Could say wait through next scheduled run and check one start. Need maybe "Do not break either mechanism": - We are not disabling `crond` or `systemd`; only duplicate job entry removed. If you must keep both entries temporarily, lock prevents simultaneous duplicate. But if schedules at different times, you must disable one from triggering. Need maybe include "If the cron entry is in `/etc/cron.daily`, removing a file there means cron will no longer run it; systemd timer remains. Test with `run-parts --test /etc/cron.daily`? Not necessary. Let's think if there is a "same plus" context maybe previous answer was about "application deployment validation, log rotation, backups" maybe user expects "same format". Without context, answer robust. Need maybe discuss "verify double-fire risk" more than just registrations. Could include "If systemd timer's next run matches cron schedule, confirm by using `systemd-analyze calendar` and perhaps `croniter`." Maybe a sample: Cron: ``` 15 4 * * * ``` Systemd: ``` OnCalendar=*-*-* 04:15:00 ``` `systemd-analyze calendar '*-*-* 04:15:00'` yields next. But no need. Maybe include a "sample output to look for": ``` systemctl list-timers: NEXT ... LEFT ... ACTIVATES Tue 2025-... 03:30 ... myjob.service cron: 30 3 * * * appuser /usr/local/bin/myjob ``` This is clear. Need maybe mention "cron often has 60s granularity; systemd default AccuracySec=1min means both will fire within same minute but not exact same second. `flock` handles." Need maybe include "If systemd timer has `Persistent=true`, it may fire at boot after downtime while cron also fires at its scheduled time; decide whether catch-up is desired. If not, set Persistent=false." This is valuable for double-fire. Add in verification/fix: - "Check `Persistent=`. If the job is not safe to run catch-up, set `Persistent=false`." But if both are same schedule and server was down at scheduled time, cron generally doesn't catch up (unless anacron), systemd timer with `Persistent=true` will. This isn't simultaneous but extra run after downtime. User asks double-fire risk after deploy. Maybe include in nuances. Need maybe mention "anacron" because cron daily may catch up, and systemd `Persistent` too; double catch-up. Check `/etc/anacrontab` if cron.daily/anacron. Could include a short "extra checks" paragraph: - Compare user, command, schedule, timezone, catch-up (`@reboot`, anacron, `Persistent=true`). These can create duplicates even without identical minute. Need maybe mention "job-level idempotency": In addition to scheduler dedupe, make job idempotent/transactional; use DB unique key or lease. But maybe too much. Let's think of exact wording: "Verify and resolve double-fire risk without breaking either mechanism" Could also be interpreted as "both cron and systemd timer register the same job; verify and resolve risk without breaking either mechanism" Maybe they want "Use `systemd-run` to generate? `systemctl mask`?" No, "without breaking either mechanism" maybe "don't remove cron entirely or disable systemd; just prevent double by setting one as the source of truth with lock." But then "state which you would remove" means remove duplicate registration from one. We'll phrase: "First add a shared lock so both can coexist safely during rollout. Then remove the duplicate scheduler registration. Removing the cron job does not break cron the mechanism; it just stops cron from owning this job." Need maybe "state which would remove and why" in final direct sentence: "I would remove the cron registration and keep the systemd timer." Then reasons. Let's consider if "systemd timer" maybe is not "systemd cron" but "systemd-tmpfiles" etc. No. Potential final answer should include "don't use `systemctl disable --now myjob.timer` if you keep it; if you remove cron." Maybe if someone chooses remove systemd, commands: ``` systemctl disable --now myjob.timer systemctl daemon-reload ``` But not necessary; maybe include exception. Need maybe "after deploy" "cron and systemd timer BOTH register the same job" could happen because package has two subpackages installed. Use package manager: - On Debian: ``` dpkg -S /etc/cron.d/myjob apt-get purge myjob-cron ``` - On RHEL: ``` rpm -qf /etc/cron.d/myjob dnf remove myjob-cron ``` But if same package installs both, can't remove package without timer; modify package. If separate, remove cron subpackage. We'll include in a small note: "If `/etc/cron.d/myjob` is owned by a package, remove that package or adjust deployment; deleting the file alone will be reintroduced by next deploy." Need maybe "Verify service user matches cron user." Systemd: ``` systemctl show myjob.service -p User -p Group -p ExecStart ``` Cron in `/etc/cron.d` has user field; user crontab owner. If different users, lock file permissions can be misconfigured. Include in verify: ``` systemctl show myjob.service -p User -p Group -p ExecStart ``` Cron: - user field or crontab owner. Good. Need maybe "systemd timer service may have `ConditionPathExists` or `ExecCondition` to disable when cron has run"? Hmm. Could resolve double-fire by adding `ExecCondition=/usr/bin/flock -n`? Better not. Potential alternative: Use systemd timer to run "cron job wrapper" but remove cron line. Not relevant. Let's maybe include a "minimal one-liner lock" in final for people not wanting wrapper: ``` /usr/bin/flock -n -E 0 /var/lib/myjob/lock /usr/local/bin/myjob ``` Then mention wrapper logs skip. The final can use wrapper as main. Need maybe be careful about "both register the same job" and "double-fire" can be a functional bug due to deploy not idempotent. The proper code/deploy change: - Add a feature flag `scheduler: systemd` to deploy; stop installing cron. - "Make install script idempotent and mutually exclusive: if install_systemd_timer then remove/disable cron; if install_cron then remove timer." Include maybe. Let's think if there is a standard "double registration" due to `dpkg` triggers and systemd preset: - If a package ships both `/etc/cron.d/foo` and `/usr/lib/systemd/system/foo.timer`, package manager enables systemd timer by preset. Remove cron file from package or use conffile. If installed from deploy, ensure uninstall old scheduler. - For config management, adding `state: absent` for cron entry. Maybe answer can mention "Do not just comment out cron manually; fix deploy." Good. Need maybe include "cron doesn't have per-job locking; systemd `RefuseManualStart` no; `flock` is standard." Potential command for verifying cron daemon active: ``` systemctl is-active cron crond ``` If cron not active, registration inert. But prompt says cron register, maybe active. Add: ``` systemctl is-active cron || systemctl is-active crond ``` Cron service name varies. Need "if active." Also check systemd timer active. Use: ``` systemctl is-enabled myjob.timer; systemctl is-active myjob.timer ``` For cron: ``` systemctl is-active cron || systemctl is-active crond ``` Cron spool entry maybe present but cron daemon not active; then no double. Include maybe. Need maybe include "systemd timer units in `/etc/systemd/system` or `/usr/lib/systemd/system`; if multiple unit files with same name, check `systemctl cat` picks the right one. `systemd-delta` maybe." Could be too much. Need maybe "systemctl list-timers --all" includes timer and schedule. Good. Let's decide final structure: ``` Assumption... ## Verify Commands... What proves risk... ## Resolve safely 1. Add shared lock 2. Test lock 3. Remove duplicate scheduler registration ## Which to remove Keep systemd timer; remove cron; reasons; exceptions. ``` Let's write final answer with code blocks but not too long. Need ensure commands are correct. Let's refine command examples: Systemd verification: ``` systemctl status myjob.timer --no-pager systemctl cat myjob.timer systemctl cat myjob.service systemctl show myjob.timer \ --property=ActiveState,Unit,LastTriggerUSec,NextElapseUSecRealtime,OnCalendar,OnBootSec,OnUnitActiveSec,Persistent,RandomizedDelaySec systemctl show myjob.service --property=User,Group,ExecStart systemd-analyze calendar "$(systemctl show myjob.timer --property=OnCalendar --value)" ``` `systemctl show` with multiple properties comma-separated? I think syntax `--property=ActiveState,Unit,...` works. To be safe, use repeated `-p`: ``` systemctl show myjob.timer -p ActiveState -p Unit ... ``` Good. Cron: ``` PAT='/usr/local/bin/myjob|myjob-run' grep -RInE "$PAT" /etc/crontab /etc/cron.d /etc/cron.hourly /etc/cron.daily /etc/cron.weekly /etc/cron.monthly /etc/anacrontab /var/spool/cron /var/spool/cron/crontabs 2>/dev/null crontab -l -u appuser 2>/dev/null | grep -nE "$PAT" ``` But if user isn't appuser, need check all user crontabs? `grep` in spool covers. Actual logs: ``` journalctl -u myjob.service --since "24 hours ago" --no-pager journalctl -t CRON --since "24 hours ago" --no-pager | grep -E "$PAT" grep -RInE "$PAT" /var/log/cron* /var/log/syslog* 2>/dev/null | tail -100 ``` Maybe `journalctl -t CRON` only captures cron messages, but systemd service logs with `-u myjob.service`. Good. Schedule equivalence: ``` systemctl list-timers myjob.timer --no-pager ``` Cron no next time. If cron expression `30 3 * * *`, systemd timer `OnCalendar=*-*-* 03:30:00`, risk. Need mention compare. Lock wrapper: ``` install -d -o appuser -g appuser -m 0750 /var/lib/myjob touch /var/lib/myjob/lock chown appuser:appuser /var/lib/myjob/lock chmod 0600 /var/lib/myjob/lock ``` If job root, substitute. But if appuser group not exist? okay. Use `appuser:appuser` not always group same; but okay. Maybe "or root:root" if root. Good. Wrapper: ``` cat >/usr/local/bin/myjob-run <<'EOF' #!/bin/sh set -u tag=myjob lock=/var/lib/myjob/lock exec 9>"$lock" if ! flock -n 9; then logger -t "$tag" "lock busy; skipping duplicate invocation" exit 0 fi logger -t "$tag" "start" /usr/local/bin/myjob rc=$? logger -t "$tag" "finish rc=$rc" exit "$rc" EOF chmod 0755 /usr/local/bin/myjob-run ``` Need if `logger` not found or fails due permissions? logger writes syslog; if logger fails with set -u only, not fatal. If logger fails, command not affected. Good. Potential `exec 9>"$lock"` if lock path unwritable, shell exits with error. Need ownership correct. Systemd: ``` [Service] ExecStart=/usr/local/bin/myjob-run ``` Cron: ``` 30 3 * * * appuser /usr/local/bin/myjob-run ``` If systemd service has `User=appuser`, okay. After changing service: ``` systemctl daemon-reload ``` Test: ``` flock /var/lib/myjob/lock -c 'sleep 30' & systemctl start myjob.service journalctl -t myjob -n 20 --no-pager ``` Potential `flock` as root can open file; `systemctl start` service as appuser can open due mode 0600. Good. Wait, `flock /var/lib/myjob/lock -c 'sleep 30'` opens file in read/write by default? If file mode 0600 owned appuser and root runs, okay. If a non-root tester runs as appuser, okay. Good. Removal: ``` # from deploy/config mgmt, not just ad hoc rm -f /etc/cron.d/myjob # or remove only exact line from /etc/crontab/shared file # or user crontab crontab -l -u appuser > /root/appuser.cron.$(date +%s).bak crontab -l -u appuser | grep -vF '/usr/local/bin/myjob-run' | crontab -u appuser - ``` Need if pipeline has no lines maybe okay. If the job command was original not wrapper, use pattern found. Better: ``` crontab -l -u appuser | grep -vE "$PAT" | crontab -u appuser - ``` But `PAT` includes `/usr/local/bin/myjob|myjob-run`. Good. However, if crontab has comments with job name, they'd remain; fine. Package: ``` dpkg -S /etc/cron.d/myjob 2>/dev/null || rpm -qf /etc/cron.d/myjob ``` Then remove appropriate package/subpackage. If same package owns both, patch package/deploy. Good. After removal: ``` systemctl enable --now myjob.timer systemctl restart myjob.timer # optional after unit changes systemctl list-timers myjob.timer --no-pager grep -RInE "$PAT" /etc/crontab /etc/cron.d /var/spool/cron /var/spool/cron/crontabs 2>/dev/null ``` Maybe `systemctl restart myjob.timer` can cause immediate service? Usually no, but timer restart might invoke? Actually restarting a timer unit may cause it to become active and may trigger if elapsed? Not typically. For safe, use: ``` systemctl daemon-reload systemctl enable --now myjob.timer ``` If timer was active before, no restart. If changed timer spec, use `systemctl restart` per docs. But "without breaking" maybe avoid. We'll say: ``` # if timer unit changed systemctl daemon-reload systemctl restart myjob.timer ``` Maybe "restart" not necessary for service ExecStart change; daemon-reload is enough. `systemctl restart myjob.timer` might reset timer state. Fine. Need maybe "If both must remain long term, you can leave the lock and set only one to actually schedule; e.g., deploy the other with `state: absent` or disabled. Do not rely on the lock to prevent duplicate non-overlapping executions." Need maybe "which remove and why" final: ``` Default: remove cron job, keep systemd timer. Rationale... Exception... ``` Potential reasons: - `systemctl list-timers`/`journalctl` clearer than parsing cron logs. - Systemd service gives explicit User/Environment, dependencies, network/mount readiness, sandboxing/resource limits, timeouts, failure policies. - `Persistent`, `RandomizedDelaySec`, `AccuracySec` better for catchup/load-spreading. - Unit files deploy atomically and can be version-controlled; cron entries scattered and can be silently overwritten by crontab. - cron should remain for legacy jobs; don't disable cron daemon. Maybe "cron is simple and portable, but since systemd timer is already present and target systems use systemd, keep systemd." Need maybe "If the organization has standardized on cron, remove systemd timer instead. In that case: ``` systemctl disable --now myjob.timer ``` and delete unit files from deploy, keep cron. But I would not choose that on modern systemd Linux." Could include. Need maybe "verify no double-fire after removal": - After next scheduled run: ``` journalctl -t myjob --since "last scheduled run" | grep -E 'start|finish|lock busy' journalctl -t CRON --since "last scheduled run" | grep -E "$PAT" ``` Expect exactly one start/finish and no CRON execution. Potential issue with `logger -t myjob`: If `journalctl -t myjob` logs from both cron and systemd; good. Without logger, job logs may be in service journal and cron mail? But wrapper logs. Need maybe "cron and systemd might not run at exact same second; `flock` needs to cover entire job duration, including spawned children." Add a warning: ``` The lock only works if the real command runs in the foreground and all work completes before wrapper exits. If it daemonizes/forks and exits, fix that or use a lease/PID file; otherwise a later scheduler can run while children still operate. ``` This is useful. Need maybe "If cluster-wide job, use DB advisory lock/Redis Redlock; `flock` is per-host." Add small note. Let's ensure final answer doesn't overdo. Desired oververbosity 5; answer can be around 1000-1500 words? Maybe too long. Need probably 800-1200. Let's aim ~1000. But user asks detailed commands; okay. Let's draft final: ``` Assume names... ``` Need use "Same, plus" maybe maybe they want direct continuation. We can start "Assuming the job is `myjob`..." Good. Potential final: ``` First, don't disable `crond` or `systemd` globally. Only the duplicate job registration should be removed. ``` Good. Let's write: ``` PAT='myjob|/usr/local/bin/myjob' ... ``` Need quoting for grep: ``` grep -RInE "$PAT" ... ``` If `PAT` includes pipe, okay. Potential command if `systemd-analyze calendar "$(systemctl show ...)"` returns blank; if blank skip. Maybe not. Let's include "What you are looking for": - same command/arguments - same effective user - overlapping trigger time/timezone/catch-up - both active Add: ``` systemctl is-enabled myjob.timer systemctl is-active myjob.timer systemctl is-active cron || systemctl is-active crond ``` Maybe include. Let's produce final with sections. Need think if "state which you would remove and why" maybe maybe should state upfront and not bury. We can put at top: "Resolution: add lock as temporary guard, then remove cron entry and keep systemd timer." Then details. Let's draft final answer now. But before final, let's consider alternative answer "remove systemd timer" maybe could be argued because cron is legacy and more portable. Which is more likely expected? The prompt says "cron and a systemd timer BOTH register the same job" in modern Linux. The standard recommendation is keep systemd timer. Need not waffle. State default remove cron. Could there be a specific "double-fire risk without breaking either mechanism" maybe in an "after deploy" scenario with both "cron and systemd timer" in "infrastructure" where "cron" has no systemd timer features. The correct "which remove" maybe "remove cron, because systemd timers offer RandomizedDelaySec and flock etc." Good. Need maybe "without breaking either mechanism" maybe they don't want to remove the scheduler package; "remove the duplicate registration in cron, not cron itself." We'll explicitly say. Potential "verify" maybe "Check systemd timers and cron jobs via commands; check logs for double invocation; add lock; remove one." Good. Let's maybe include "if `Persistent=true`, decide catch-up semantics; set false if duplicate after downtime is dangerous." Where to put: In verification: ``` Also inspect Persistent=true and anacron/@reboot; these can produce extra catch-up runs even if cron and systemd don't fire the same minute. ``` In fix: ``` If catch-up is not desired, set Persistent=false; if desired, keep it but only one scheduler owns catch-up. ``` Good. Potential "actual double-fire" if both at same time but lock prevents; after removal, no double. Fine. Let's final now. Assuming the job is `myjob`, with `myjob.timer` / `myjob.service` and command `/usr/local/bin/myjob`, do this: ## 1. Verify both registrations and actual firing risk ```bash PAT='myjob|/usr/local/bin/myjob' # Systemd systemctl list-timers --all --no-pager | grep -E "$PAT" systemctl cat myjob.timer systemctl cat myjob.service systemctl show myjob.timer \ -p ActiveState -p Enabled \ -p OnCalendar -p OnBootSec -p OnUnitActiveSec \ -p Persistent -p RandomizedDelaySec \ -p NextElapseUSecRealtime -p LastTriggerUSec systemctl show myjob.service -p User -p Group -p ExecStart # Cron: distro paths vary grep -RInE "$PAT" \ /etc/crontab \ /etc/cron.d \ /etc/cron.hourly \ /etc/cron.daily \ /etc/cron.weekly \ /etc/cron.monthly \ /etc/anacrontab \ /var/spool/cron \ /var/spool/cron/crontabs 2>/dev/null # Current user's crontab, if relevant crontab -l | grep -nE "$PAT" ``` Check logs for evidence of duplicate invocation: ```bash journalctl -u myjob.service --since "24 hours ago" --no-pager journalctl -t CRON --since "24 hours ago" --no-pager | grep -E "$PAT" ``` Also compare: - same command/arguments, - same effective user, - same schedule/timezone, - boot-time or catch-up behavior (`@reboot`, anacron, `Persistent=true`). If cron and systemd both call the same command on overlapping schedules, double-fire risk is confirmed. ## 2. Protect against double execution without disabling either mechanism Before removing one registration, put both paths behind a shared lock. Example wrapper: ```bash cat > /usr/local/bin/myjob-run <<'EOF' #!/bin/sh set -u tag=myjob lock=/var/lib/myjob/lock exec 9>"$lock" if ! flock -n 9; then logger -t "$tag" "lock busy; skipping duplicate invocation" exit 0 fi logger -t "$tag" "start" /usr/local/bin/myjob rc=$? logger -t "$tag" "finish rc=$rc" exit "$rc" EOF chmod 0755 /usr/local/bin/myjob-run ``` Create the lock file with ownership matching the job user, e.g. `appuser`: ```bash install -d -o appuser -g appuser -m 0750 /var/lib/myjob touch /var/lib/myjob/lock chown appuser:appuser /var/lib/myjob/lock chmod 0600 /var/lib/myjob/lock ``` Use the same wrapper from both mechanisms. Cron: ```cron 30 3 * * * appuser /usr/local/bin/myjob-run ``` Systemd service: ```ini [Service] ExecStart=/usr/local/bin/myjob-run ``` Then test: ```bash systemctl daemon-reload # Hold the lock and try the systemd path flock /var/lib/myjob/lock -c 'sleep 30' & systemctl start myjob.service journalctl -t myjob -n 20 --no-pager ``` You should see `lock busy; skipping duplicate invocation`. Important: this lock prevents overlapping execution. It does not make two independent schedules correct. If cron fires at 03:30 and systemd fires at 03:35, you can still run the job twice. The permanent fix is to remove one scheduler registration. ## 3. Remove the duplicate registration I would remove the cron entry and keep the systemd timer. Do not disable `cron`/`crond` globally unless it has no other purpose; remove only this job’s cron registration. Example: ```bash rm -f /etc/cron.d/myjob ``` Or, if it is in a user crontab: ```bash crontab -l -u appuser > /root/appuser.crontab.bak crontab -l -u appuser | grep -vE 'myjob|/usr/local/bin/myjob' | crontab -u appuser - ``` If the cron file was installed by a package or deployment tool, fix the package/deploy manifest so the next deploy does not recreate it: ```bash dpkg -S /etc/cron.d/myjob 2>/dev/null || rpm -qf /etc/cron.d/myjob ``` Then ensure the systemd timer is the only active scheduler: ```bash systemctl daemon-reload systemctl enable --now myjob.timer systemctl list-timers myjob.timer --no-pager ``` Verify cron no longer references the job: ```bash grep -RInE 'myjob|/usr/local/bin/myjob' \ /etc/crontab /etc/cron.d /var/spool/cron /var/spool/cron/crontabs 2>/dev/null ``` ## Which I would remove and why I would remove the cron registration and keep the systemd timer. Reasons: - systemd gives better observability: `systemctl list-timers`, `systemctl status`, `journalctl`. - systemd service units give explicit control over `User`, `Environment`, timeouts, dependencies, resource limits, and sandboxing. - systemd timers support useful features such as `Persistent`, `RandomizedDelaySec`, `AccuracySec`, and boot/network/mount dependencies. - Unit files are easier to deploy, review, and version-control consistently. - Keeping only one scheduler makes ownership and failure handling clear. Exception: if the fleet is non-systemd, containers without systemd, or the organization has standardized on cron, then remove the systemd timer instead and keep cron. On a normal systemd Linux host, though, keep the systemd timer and remove the cron job.

Same, plus: after deploy, cron and a systemd timer BOTH regi...