Things to watch out for when using cron with Laravel's scheduler at second-level intervals

I really struggled for 10 hours trying to solve this problem. To put it simply. ```php Schedule::command(TestCommand::class)->everyMinute()->onOneServer()->runInBackground(); Schedule::command(Test2Command::class)->everyTwoSeconds()->onOneServer()->runInBackground(); Schedule::command(Test2Command::class)->everyTwoSeconds()->onOneServer()->runInBackground(); ``` Sometimes you configure the Laravel scheduler to run at second-level intervals like this. The documentation says to do this: ```sh SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin * * * * * php /var/www/artisan schedule:run > /tmp/cron.last 2>&1 ``` It says to set it up like this. But there's one major problem here. schedule:run starts a process, and that process repeatedly runs every second for one minute. That's how the code is implemented in Laravel. My first thought was: is this an onOneServer issue? A runInBackground issue? I debugged command? job? and those areas, but the problem wasn't there. So I checked whether PHP itself was running. It wasn't even starting at all. At first, I was installing dcron on an Alpine Linux image and using that. I wondered if dcron was the problem, so I removed it and tried switching to BusyBox cron. But even after switching to BusyBox cron, the problem still wasn't fixed. So eventually, I enabled logging for cron. ```sh cat cron.log crond: crond (busybox 1.37.0) started, log level 8 crond: USER cron pid 86 cmd php /var/www/artisan schedule:run > /tmp/cron.last 2>&1 crond: USER cron pid 87 cmd echo "Cron is working: $(date)" >> /tmp/cron_test.log crond: user cron: process already running: php /var/www/artisan schedule:run > /tmp/cron.last 2>&1 crond: USER cron pid 896 cmd echo "Cron is working: $(date)" >> /tmp/cron_test.log crond: USER cron pid 904 cmd php /var/www/artisan schedule:run > /tmp/cron.last 2>&1 crond: USER cron pid 905 cmd echo "Cron is working: $(date)" >> /tmp/cron_test.log ``` This is what I got. `crond: user cron: process already running` It wasn't running because another process was already running... Oh... I can't believe I'm only figuring this out now 😭 So I looked into it and found: I found that it checks exactly like this: if a process is already running, it doesn't start another one. Seriously... I learned that cron prevents processes with duplicate PIDs from running. Or did I already know that and forget? ㅡ. ㅡ ? Ah... so the way I fixed it was... ```sh SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin * * * * * php /var/www/artisan schedule:run > /tmp/cron.last 2>&1 & ``` Adding & at the end changed it to run in the background, which solved the problem. I used to add & at the end, but when the second-level scheduler was added recently, I followed the documentation and removed &. I think that's what caused this problem. So why does this problem happen? php schedule:run needs to terminate by the 59-second mark. ```php protected function repeatEvents($events) { $hasEnteredMaintenanceMode = false; while (Date::now()->lte($this->startedAt->endOfMinute())) { foreach ($events as $event) { if ($this->shouldInterrupt()) { return; } if (! $event->shouldRepeatNow()) { continue; } $hasEnteredMaintenanceMode = $hasEnteredMaintenanceMode || $this->laravel->isDownForMaintenance(); if ($hasEnteredMaintenanceMode && ! $event->runsInMaintenanceMode()) { continue; } if (! $event->filtersPass($this->laravel)) { $this->dispatcher->dispatch(new ScheduledTaskSkipped($event)); continue; } if ($event->onOneServer) { $this->runSingleServerEvent($event); } else { $this->runEvent($event); } $this->eventsRan = true; } Sleep::usleep(100000); } } ``` But even if this is set to run every 2 seconds, it doesn't run at 2, 4, 6, 8 seconds. There's a slight delay each time a command is read, and when that happens, the schedule can slip and run at times like 2, 4, 7, 9, 11, 12, 14. So if the commands that run at 59 seconds don't all finish within one second, the process should terminate at 59 seconds, but because the commands started at 59 seconds haven't finished yet, cron will encounter the `crond: user cron: process already running` problem. Ultimately, when running the scheduler, it's a given that you must always use `runInBackground()`. If you're using `schedule:run`, you need to add & at the end to run it in the background. You need to manage overlaps in your code. ```php Schedule::command(Test2Command::class)->everyTwoSeconds()->onOneServer()->withoutOverlapping()->runInBackground(); ``` By using `withoutOverlapping()`. For reference, whether you're using jobs for everything or dispatching commands and passing them to jobs, once you have 40 or 50 commands, there's no guarantee that the process will stay under 60 seconds. Of course, this may be an issue specific to BusyBox cron or dcron. It may also have started when I switched to an Alpine Linux image. Anyway, if you're using `schedule:run` to run the scheduler, I recommend running it in the background.

More posts