Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory leaks in handling updates loop #1485

Closed
vtsykun opened this issue Feb 18, 2024 · 5 comments
Closed

Memory leaks in handling updates loop #1485

vtsykun opened this issue Feb 18, 2024 · 5 comments

Comments

@vtsykun
Copy link

vtsykun commented Feb 18, 2024

I noticed large memory leaks when processing a large number of messages. After investigation I found out that these are related to Peer, Username, FileReference database.
So I created a special a Least Recently Used (LRU) MemoryArray DbArray adapter with limited size. This partially solved the problem, memory usage reduced from +500 MB to 60 MB after restart, but there are still memory leaks

2024-02-17 06:41:10.849338 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 62.64 MB, PHP Real 44 MB, now 40.14 MB, peak 50 MB
2024-02-17 06:56:36.449571 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 67.48 MB, PHP Real 48 MB, now 35.96 MB, peak 52 MB
2024-02-17 07:12:02.050498 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 66.29 MB, PHP Real 46 MB, now 37.74 MB, peak 52 MB
2024-02-17 07:27:27.651332 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 66.34 MB, PHP Real 46 MB, now 40.2 MB, peak 54 MB
2024-02-17 07:42:53.252431 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 68.43 MB, PHP Real 48 MB, now 42.51 MB, peak 56 MB
2024-02-17 07:58:18.852877 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 70.45 MB, PHP Real 50 MB, now 44.72 MB, peak 56 MB
2024-02-17 08:13:44.454213 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 72.06 MB, PHP Real 52.5 MB, now 47.78 MB, peak 60.5 MB
2024-02-17 08:29:10.055421 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 74.29 MB, PHP Real 54.5 MB, now 50.68 MB, peak 62.5 MB
2024-02-17 08:44:35.656697 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 76.52 MB, PHP Real 56.5 MB, now 53.29 MB, peak 64.5 MB
2024-02-17 09:00:01.256927 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 78.92 MB, PHP Real 58.5 MB, now 50.1 MB, peak 66.5 MB
2024-02-17 09:15:26.856792 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 81.16 MB, PHP Real 60.5 MB, now 53.36 MB, peak 66.5 MB
2024-02-17 09:30:52.456841 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 83.9 MB, PHP Real 65 MB, now 58.55 MB, peak 73 MB
2024-02-17 09:46:18.057578 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 86.16 MB, PHP Real 67 MB, now 61.33 MB, peak 75 MB
2024-02-17 10:01:43.658044 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 88.35 MB, PHP Real 69 MB, now 63.51 MB, peak 77 MB
2024-02-17 10:17:09.258085 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 90.57 MB, PHP Real 71 MB, now 65.88 MB, peak 79 MB
2024-02-17 10:32:34.858481 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 92.77 MB, PHP Real 73 MB, now 68.2 MB, peak 81 MB
2024-02-17 10:48:00.459272 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 95 MB, PHP Real 75 MB, now 70.74 MB, peak 83 MB
2024-02-17 11:03:26.060398 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 97.05 MB, PHP Real 77 MB, now 67.43 MB, peak 83 MB
2024-02-17 11:18:51.661908 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 97.74 MB, PHP Real 79 MB, now 70.55 MB, peak 85 MB
2024-02-17 11:34:17.262058 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 101.57 MB, PHP Real 81 MB, now 73.18 MB, peak 87 MB
2024-02-17 11:49:42.863397 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 104.82 MB, PHP Real 88 MB, now 80.95 MB, peak 96 MB
2024-02-17 12:05:08.463539 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 107.03 MB, PHP Real 90 MB, now 83.23 MB, peak 96 MB
2024-02-17 12:20:34.063989 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 109.24 MB, PHP Real 92 MB, now 85.53 MB, peak 98 MB
2024-02-17 12:35:59.664383 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 111.51 MB, PHP Real 94 MB, now 88.59 MB, peak 102 MB
2024-02-17 12:51:25.265648 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 113.62 MB, PHP Real 96 MB, now 84.8 MB, peak 102 MB
2024-02-17 13:06:50.865821 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 113.83 MB, PHP Real 96 MB, now 86.91 MB, peak 102 MB
2024-02-17 13:22:16.466524 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 116.01 MB, PHP Real 98 MB, now 89.1 MB, peak 104 MB
2024-02-17 13:37:42.066671 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 118.12 MB, PHP Real 100 MB, now 91.43 MB, peak 106 MB
2024-02-17 13:53:07.666765 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 118.37 MB, PHP Real 100 MB, now 93.23 MB, peak 108 MB
2024-02-17 14:08:33.267200 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 120.57 MB, PHP Real 102 MB, now 95.54 MB, peak 110 MB
2024-02-17 14:23:58.866784 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 122.72 MB, PHP Real 104 MB, now 97.44 MB, peak 112 MB
2024-02-17 14:39:24.467094 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 124.93 MB, PHP Real 106 MB, now 93.29 MB, peak 112 MB
2024-02-17 14:54:50.066821 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 125.07 MB, PHP Real 106 MB, now 95.34 MB, peak 112 MB
2024-02-17 15:10:15.666966 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 125.36 MB, PHP Real 106 MB, now 97.39 MB, peak 114 MB
2024-02-17 15:25:41.267423 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 127.5 MB, PHP Real 108 MB, now 99.06 MB, peak 114 MB
2024-02-17 15:41:06.868354 docker-entrypoint, xxxxxxxxx: 	Memory usage check: RSS 129.65 MB, PHP Real 110 MB, now 100.97 MB, peak 118 MB

Then I tried to debug with https://github.com/BitOne/php-meminfo, but seems this library not dump Fibers memory stack dump

\Revolt\EventLoop::onSignal(\SIGUSR1, function () {
    echo "\n---------\n\n\n-SIGUSR1-\n\n\n";
    $file = is_writable("/data") ? '/data/mem_dump.json' : '/tmp/mem_dump.json';
    if (\file_exists($file)) {
        @\unlink($file);
    }
    \meminfo_dump(\fopen($file, 'w'));
});

What places in the code can I check, (for example where local caching is used) to fix memory leaks?
@Teneja
Copy link

Teneja commented Feb 18, 2024 via email

@danog
Copy link
Owner

danog commented Feb 18, 2024

You're using the event handler, and the latest v8 beta, correct?
The behavior of the DbArray is not a memory leak, it is the intended behavior, caching database entries into RAM for the period of time specified in the settings.

This behavior can be turned off altogether by specifying a TTL of 0 in the settings, at the expense of performance, which is something I do not recommend doing.

@vtsykun
Copy link
Author

vtsykun commented Feb 19, 2024 •
edited
Loading

Hi @danog I found it. Problem in MinDatabase cache

final class MinDatabase implements TLCallback
{
    /**
     * Temporary cache during deserialization.
     */
    private array $cache = [];

I use my own fork for debug memory https://github.com/vtsykun/php-meminfo where fixed bugs BitOne/php-meminfo#133 BitOne/php-meminfo#135

$ php bin/analyzer top-children tmp/mem_dump.json0
+-----+----------------+----------+
| Num | Item ids       | Children |
+-----+----------------+----------+
| 1   | 0x7fd2d6099a48 | 120357   |
| 2   | 0x7fd2d3cdce80 | 6391     |
| 3   | 0x7fd2d604e8e8 | 1472     |
| 4   | 0x7fd2d6d48c98 | 1301     |
| 5   | 0x7fd2d6d48c88 | 1195     |
+-----+----------------+----------+



Path from 0x7fd2dad5ae00
+------------------------+
| Id: 0x7fd2d6099a48     |
| Type: array            |
| Size: 72 B             |
| Is root: No            |
| Children count: 120357 |
+------------------------+
         ^          
         |          
       cache        
         |          
         |          
+-----------------------------------------------------+
| Id: 0x7fd2d7fc7820                                  |
| Type: object                                        |
| Class: danog\MadelineProto\MTProtoTools\MinDatabase |
| Object Handle: 164                                  |
| Size: 72 B                                          |
| Is root: No                                         |
| Children count: 6                                   |
+-----------------------------------------------------+
         ^          
         |          
    minDatabase     
         |          
         |          
+------------------------------------+
| Id: 0x7fd2d3de9800                 |
| Type: object                       |
| Class: danog\MadelineProto\MTProto |
| Object Handle: 159                 |
| Size: 72 B                         |
| Is root: No                        |
| Children count: 78                 |
+------------------------------------+

@danog
Copy link
Owner

danog commented Feb 20, 2024

Thank you for investigating and finding the issue, fixed in the latest commit!

@danog danog closed this as completed Feb 20, 2024
@vtsykun
Copy link
Author

vtsykun commented Feb 20, 2024

Hi @danog it's works now.

Thank you for the fast fix and nice software.

2024-02-20 11:55:50.785497 	Memory usage: RSS 72.50 MB, PHP Heap 50 MB, mean 40.26 MB, peak 56 MB
2024-02-20 12:11:16.433019 	Memory usage: RSS 75.43 MB, PHP Heap 54 MB, mean 43.60 MB, peak 56 MB
2024-02-20 12:26:42.033885 	Memory usage: RSS 72.62 MB, PHP Heap 50 MB, mean 41.31 MB, peak 56 MB
2024-02-20 12:42:07.633855 	Memory usage: RSS 72.64 MB, PHP Heap 50 MB, mean 42.71 MB, peak 56 MB
2024-02-20 12:57:33.233858 	Memory usage: RSS 72.62 MB, PHP Heap 50 MB, mean 43.31 MB, peak 56 MB
2024-02-20 13:12:58.834414 	Memory usage: RSS 72.63 MB, PHP Heap 50 MB, mean 43.73 MB, peak 56 MB
2024-02-20 13:28:24.434696 	Memory usage: RSS 72.63 MB, PHP Heap 50 MB, mean 44.66 MB, peak 56 MB
2024-02-20 13:43:50.034722 	Memory usage: RSS 72.64 MB, PHP Heap 50 MB, mean 45.24 MB, peak 56 MB
2024-02-20 13:59:15.635520 	Memory usage: RSS 72.63 MB, PHP Heap 50 MB, mean 38.67 MB, peak 56 MB
2024-02-20 14:14:41.236329 	Memory usage: RSS 72.63 MB, PHP Heap 50 MB, mean 39.54 MB, peak 56 MB
2024-02-20 14:30:06.836919 	Memory usage: RSS 72.63 MB, PHP Heap 50 MB, mean 40.18 MB, peak 56 MB
2024-02-20 14:45:32.436505 	Memory usage: RSS 72.64 MB, PHP Heap 50 MB, mean 40.60 MB, peak 56 MB

@sj-i sj-i mentioned this issue Feb 20, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@danog @vtsykun @Teneja