Server-reload() does not terminate Swoole Timers, gracefully

[fakharksa]

1. What did you do? If possible, provide a simple script for reproducing the error.

Terminology used in Code below:

• class prefix sw stands for swoole\ and osw for openswoole. so, swTimer is Swoole\Timer.
• $this->swoole_ext == 1 evaluates to true means "swoole" extension is enabled, otherwise "openswoole" extension is enabled.

I create this Swoole's timer from inside onMessage Event and in the same onMEssage, i also calll reload(), as below:

                if ($frame->data == 'reload-code') {
                    if ($this->swoole_ext == 1) {
                        echo PHP_EOL.'In Reload-Code: Clearing All Swoole-based Timers'.PHP_EOL;
                        swTimer::clearAll();
                    } else {
                        echo PHP_EOL.'In Reload-Code: Clearing All OpenSwoole-based Timers'.$fd.PHP_EOL;
                        oswTimer::clearAll();
                    }
//                    self::$fds = null;
//                    unset($frame);
                    echo "Reloading Code Changes (by Reloading All Workers)".PHP_EOL;
                    $webSocketServer->reload();
                } else {
                    include_once __DIR__ . '/Controllers/WebSocketController.php';

                    global $app_type_database_driven;
                    if ($app_type_database_driven) {
                        $sw_websocket_controller = new WebSocketController($webSocketServer, $frame, $this->dbConnectionPools[$webSocketServer->worker_id]);
                    } else {
                        $sw_websocket_controller = new WebSocketController($webSocketServer, $frame);
                    }

                    $timerTime = $_ENV['SWOOLE_TIMER_TIME1'];
                    if ($this->swoole_ext == 1) {
                        self::$fds[$frame->fd][] = swTimer::tick($timerTime, $respond, $webSocketServer, $frame, $sw_websocket_controller);
                    } else {
                        self::$fds[$frame->fd][] = oswTimer::tick($timerTime, $respond, $webSocketServer, $frame, $sw_websocket_controller);
                    }
                } 

I use WebSocket's push() function from inside Swoole's Timer's callback which pushes data, and is defined as below:

        $respond = function($timerId, $webSocketServer, $frame, $sw_websocket_controller) {
            if (isset($frame->fd) && isset(self::$fds[$frame->fd])) { // if the user / fd is connected then push else clear timer.
                if ($frame->data) { // when a new message arrives from connected client with some data in it
                    $bl_response = $sw_websocket_controller->handle();
                    $frame->data = false;
                } else {
                    $bl_response = 1;
                }

                $webSocketServer->push($frame->fd,
                    json_encode($bl_response),
                    WEBSOCKET_OPCODE_TEXT,
                    SWOOLE_WEBSOCKET_FLAG_FIN); // SWOOLE_WEBSOCKET_FLAG_FIN OR OpenSwoole\WebSocket\Server::WEBSOCKET_FLAG_FIN

            } else {
                echo "Inside Event's Callback: Clearing Timer ".$timerId.PHP_EOL;
                if ($this->swoole_ext == 1) {
                    swTimer::clear($timerId);
                } else {
                    oswTimer::clear($timerId);
                }
            }
        };

Other related Code:

• Below is my onClose:

        $this->server->on('close', function($server, $fd, $reactorId) {
            echo PHP_EOL."client {$fd} closed in ReactorId:{$reactorId}".PHP_EOL;

            if ($this->swoole_ext == 1) {
                if (isset(self::$fds[$fd])) {
                    echo PHP_EOL.'On Close: Clearing Swoole-based Timers for Connection-'.$fd.PHP_EOL;
                    $fd_timers = self::$fds[$fd];
                    foreach ($fd_timers as $fd_timer){
                        if (swTimer::exists($fd_timer)) {
                            echo PHP_EOL."In Connection-Close: clearing timer: ".$fd_timer.PHP_EOL;
                            swTimer::clear($fd_timer);
                        }
                    }
                }
            } else {
                if (isset(self::$fds[$fd])) {
                    echo PHP_EOL.'On Close: Clearing OpenSwoole-based Timers for Connection-'.$fd.PHP_EOL;
                    $fd_timers = self::$fds[$fd];
                    foreach ($fd_timers as $fd_timer){
                        if (oswTimer::exists($fd_timer)) {
                            echo PHP_EOL."In Connection-Close: clearing timer: ".$fd_timer.PHP_EOL;
                            oswTimer::clear($fd_timer);
                        }
                    }
                }
            }
            unset(self::$fds[$fd]);
        });

• Below is my onDisconnect:

        $this->server->on('disconnect', function(Server $server, int $fd) {
            echo "connection disconnect: {$fd}\n";
            if ($this->swoole_ext == 1) {
                if (isset(self::$fds[$fd])) {
                    echo PHP_EOL.'On Disconnect: Clearing Swoole-based Timers for Connection-'.$fd.PHP_EOL;
                    $fd_timers = self::$fds[$fd];
                    foreach ($fd_timers as $fd_timer){
                        if (swTimer::exists($fd_timer)) {
                            echo PHP_EOL."In Disconnect: clearing timer: ".$fd_timer.PHP_EOL;
                            swTimer::clear($fd_timer);
                        }
                    }
                }
            } else {
                if (isset(self::$fds[$fd])) {
                    echo PHP_EOL.'On Disconnect: Clearing OpenSwoole-based Timers for Connection-'.$fd.PHP_EOL;
                    $fd_timers = self::$fds[$fd];
                    foreach ($fd_timers as $fd_timer){
                        if (oswTimer::exists($fd_timer)) {
                            echo PHP_EOL."In Disconnect: clearing timer: ".$fd_timer.PHP_EOL;
                            oswTimer::clear($fd_timer);
                        }
                    }
                }
            }
            unset(self::$fds[$fd]);
        });

Below are my Before/After Reload events:

As you see i tried clearing timers here but that also did not help so i commented the code.

        $this->server->on('BeforeReload', function($server)
        {
            echo "Test Statement: Before Reload". PHP_EOL;
            dump(self::$fds);
//            var_dump(get_included_files());
//            if ($this->swoole_ext == 1) {
//                if (swTimer::clearAll()) {
//                    echo PHP_EOL."Before Reload: Cleared All Swoole-based Timers".PHP_EOL;
//                } else {
//                    echo PHP_EOL."Before Reload: Could not clear Swoole-based Timers".PHP_EOL;
//                }
//            } else {
//                if (oswTimer::clearAll()) {
//                    echo PHP_EOL."Before Reload: Cleared All OpenSwoole-based Timers".PHP_EOL;
//                } else {
//                    echo PHP_EOL."Before Reload: Could not clear OpenSwoole-based Timers".PHP_EOL;
//                }
//            }
        });

        $this->server->on('AfterReload', function($server)
        {
            echo PHP_EOL."Test Statement: After Reload". PHP_EOL;
            dump(self::$fds);
//            var_dump(get_included_files());
//            if ($this->swoole_ext == 1) {
//                if (swTimer::clearAll()) {
//                    echo PHP_EOL."AfterReload: Cleared All Swoole-based Timers".PHP_EOL;
//                } else {
//                    echo PHP_EOL."AfterReload: Could not clear Swoole-based Timers".PHP_EOL;
//                }
//            } else {
//                if (oswTimer::clearAll()) {
//                    echo PHP_EOL."AfterReload: Cleared All OpenSwoole-based Timers".PHP_EOL;
//                } else {
//                    echo PHP_EOL."AfterReload: Could not clear OpenSwoole-based Timers".PHP_EOL;
//                }
//            }
        });

2. What did you expect to see?

No PHP Warning from WebSocketServer->push()

3. What did you see instead?

Inside Timers, i use push() which continues to send data to an $fd, even after Timer has been cleared. This issue occurs only if i cause $webSocketServer->reload() to be executed from other terminal.

So i get this PHP Warning, repeatedly:
PHP Warning: Swoole\WebSocket\Server::push(): session#1 does not exists in /var/www/html/swoole-serv/sw_service.php on line 425
PHP Warning: Swoole\WebSocket\Server::push(): session#1 does not exists in /var/www/html/swoole-serv/sw_service.php on line 425
PHP Warning: Swoole\WebSocket\Server::push(): session#1 does not exists in /var/www/html/swoole-serv/sw_service.php on line 425
PHP Warning: Swoole\WebSocket\Server::push(): session#1 does not exists in /var/www/html/swoole-serv/sw_service.php on line 425

4. What version of OpenSwoole are you using (show your php --ri openswoole)?

openswoole

Open Swoole => enabled
Author => Open Swoole Group <[email protected]>
Version => 22.1.2
Built => May 19 2024 22:56:05
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 3.0.2 15 Mar 2022
dtls => enabled
http2 => enabled
hook-curl => enabled
zlib => 1.2.11
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
mysqlnd => enabled
postgresql => enabled

Directive => Local Value => Master Value
openswoole.enable_coroutine => On => On
openswoole.enable_preemptive_scheduler => On => On
openswoole.display_errors => On => On
openswoole.unixsock_buffer_size => 8388608 => 8388608

5. What is your machine environment used (show your uname -a & php -v & gcc -v) ?

Linux HP-Laptop 5.17.5-76051705-generic #202204271406165150484020.04~63e51bd-Ubuntu SMP PREEMPT Wed Ma x86_64 x86_64 x86_64 GNU/Linux

PHP 8.3.8 (cli) (built: Jun 6 2024 16:58:27) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.8, Copyright (c) Zend Technologies
with Zend OPcache v8.3.8, Copyright (c), by Zend Technologies

COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1

Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.4.0-1ubuntu122.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-XeT9lY/gcc-11-11.4.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-XeT9lY/gcc-11-11.4.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu122.04)

You can also try the following OpenSwoole support channels:

• Documentation - Documentation for Open Swoole
• Slack - Slack channel of Open Swoole
• Discord - Discord server of Open Swoole

[fakharak]

Here is my GitHub Repo

The branch to checkout is named as timer_issue_branch

you can use composer install to install dependencies.

Mentioned below is the .env file:

APP_TYPE_DATABASE_DRIVEN=0
SWOOLE_DAEMONIZE=0

SWOOLE_PG_DB_DRIVER=pgsql
SWOOLE_PG_DB_ENGINE=postgres
SWOOLE_PG_DB_HOST=localhost
SWOOLE_PG_DB_PORT=5432
SWOOLE_PG_DB_DATABASE=swooledb
SWOOLE_PG_DB_USERNAME=postgres
SWOOLE_PG_DB_PASSWORD=passwd123
SWOOLE_PG_DB_KEY=pg

SWOOLE_MYSQL_DB_DRIVER=mysql
SWOOLE_MYSQL_DB_ENGINE=mysql
SWOOLE_MYSQL_DB_HOST=localhost
SWOOLE_MYSQL_DB_PORT=3306
SWOOLE_MYSQL_DB_DATABASE=muasherat_db
SWOOLE_MYSQL_DB_USERNAME=root
SWOOLE_MYSQL_DB_PASSWORD=passwd123
SWOOLE_MYSQL_DB_CHARSET=utf8mb4
SWOOLE_MYSQL_DB_KEY=mysql

DB_CONNECTION_POOL_SIZE=5

SWOOLE_TIMER_TIME1=3000

• To Start WebSocket Server:

cd to swoole-serv folder, and then run the command below
php sw_init_service.php websocket

• To send some messages from TCP Client use below:

sudo php ./websocketclient/websocketclient_usage.php

• To reload server's workers:

sudo php ./websocketclient/websocketclient_usage.php reload-code

• To Close Web Socker Server:

php sw_init_service.php close

[fakharksa]

In the screenshot below, the $server-push() continues to execute not only after $server->reload() but also after the server is safely terminated using php sw_init_service.php close
which actually causes the below executed on line 54 of sw_init_service.php

shell_exec('cd '.__DIR__.' && sudo kill -15 cat sw-heartbeat.pid && sudo rm -f sw-heartbeat.pid 2>&1 1> /dev/null&');

Here is the video link that captures my screen and i explain the issue.

About Code Flow / Organization:
In GitHub Repo's branch, i shared in my first comment

• server-side code:

sw_init_service.php (on root of project) is the entry-point in which (on line 41, 42, and 43) includes:

service_creation_params.php,
app_config.php, and ...
sw_service.php

On line 36 of service_creation_params.php, the code evaluates serverMode to SWOOLE_PROCESS when we do not specifically mention SWOOLE_BASE on command-line as second argument.

Swoole's configurations are defined in [project_root]/config/swoole_condif.php which is included in sw_service.php (the file on root of project) on line 139.

The class defined in sw_service.php contains all of the swoole events / callbacks, so it is the core / heart of the application, and it is the file where on('message, callback) is also defined, and just above onMessage is defined the callback of the Timer (the callback is assigned to $respond variable).

• client-side code
  The (TCP) client-side code is available in path below:
  [Project_Root]/websocketclient/websocketclient_usage.php

Please, let me know if you need further information to reproduce the bug.

Understanding of The Abnormal Behaviour

• Context:
  In case of $server->reload() ....

• Abnormal Behaviour
  Question-1) The Timer continously executes non-terminatingly, even after Timers have been cleared from within if ($frame->data == 'reload-code') block (Reference: Line 469 of sw_service.php), Why ? and ...

Question-2) The Timer's callback continues to execute the $server->push() even when the $server->push() is enclosed inside the if ( ... && isset(self::$fds[$frame->fd]) block which should be evaluated as false instead of true, Why false ? because after $server->reload() AND when the connection (auto) closes (through the heartbeat setting), the onClose event does NOT evaluates if (isset(self::$fds[$frame->fd]) as true which is the reason why connection is not closed from within onClose event. If so, then Why Timer's callback is still evaluating the if ( ... && isset(self::$fds[$frame->fd]) as 'true' casuing to call $server->push() continously.

[fakharksa]

I solved the issue as below.

Inside Timer's callback, instead of using
if (isset($frame->fd) && isset(self::$fds[$frame->fd][$timerId]))

i use this ...
if ($webSocketServer->exists($frame->fd))

In other words, Timer's whole callback below changes from ...

                    // This callback will be used in callback for onMessage event. next
                    $respond = function($timerId, $webSocketServer, $frame, $sw_websocket_controller) {
                        **if (isset($frame->fd) && isset(self::$fds[$frame->fd][$timerId])) { // if the user / fd is connected then push else** clear timer.
                            if ($frame->data) { // when a new message arrives from connected client with some data in it
                                $bl_response = $sw_websocket_controller->handle();
                                $frame->data = false;
                            } else {
                                $bl_response = 1;
                            }

                            $webSocketServer->push($frame->fd,
                                json_encode($bl_response),
                                WEBSOCKET_OPCODE_TEXT,
                                SWOOLE_WEBSOCKET_FLAG_FIN); // SWOOLE_WEBSOCKET_FLAG_FIN OR OpenSwoole\WebSocket\Server::WEBSOCKET_FLAG_FIN

                        } else {
                            echo "Inside Event's Callback: Clearing Timer ".$timerId.PHP_EOL;
                            if ($this->swoole_ext == 1) {
                                swTimer::clear($timerId);
                            } else {
                                oswTimer::clear($timerId);
                            }
                        }
                    };

To ...

                    // This callback will be used in callback for onMessage event. next
                    $respond = function($timerId, $webSocketServer, $frame, $sw_websocket_controller) {
                        **if ($webSocketServer->exists($frame->fd)) { // if the user / fd is connected then push else clear timer.**
                            if ($frame->data) { // when a new message arrives from connected client with some data in it
                                $bl_response = $sw_websocket_controller->handle();
                                $frame->data = false;
                            } else {
                                $bl_response = 1;
                            }

                            $webSocketServer->push($frame->fd,
                                json_encode($bl_response),
                                WEBSOCKET_OPCODE_TEXT,
                                SWOOLE_WEBSOCKET_FLAG_FIN); // SWOOLE_WEBSOCKET_FLAG_FIN OR OpenSwoole\WebSocket\Server::WEBSOCKET_FLAG_FIN

                        } else {
                            echo "Inside Event's Callback: Clearing Timer ".$timerId.PHP_EOL;
                            if ($this->swoole_ext == 1) {
                                swTimer::clear($timerId);
                            } else {
                                oswTimer::clear($timerId);
                            }
                        }
                    };

So, now when i reload the code, the Timers are cleared from inside the callback. Whereas, when code is not reloaded the timer is closed from inside the (callback for) onClose event.