-
Notifications
You must be signed in to change notification settings - Fork 705
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Loguru doesn't remove uncompressed files after compressing them #309
Comments
Hum... That's not the expected behavior. The May I ask you which |
Hi! I will close this issue in next days after test probably. I was getting this traceback in uwsgi log:
Then I found the issue #220 and there is info about enqueue param. Because my app is multiprocessing it makes sense I need the |
Ok that is bigger problem than I thought :( If I want have only one logging file for whole app instead of file for each process then I need # main.py
import os
from loguru import logger
print("pid=%s, configure logger" % os.getpid())
logger.remove()
logger.add("file.log", enqueue=True)
logger.info("Logger is configured")
from flask import Flask
application = Flask(__name__)
@application.route('/')
def hello_world():
pid = os.getpid()
print(pid)
logger.info(pid)
return str(pid) I can run ti with:
Now there are two options: First option:
|
Oh, great you figured out it was related with What do you mean by "killing the whole app"? The code you shared helped me a lot to be able to understand your issue, thanks a lot for that. However, I'm not able to reproduce the bug you're facing. If I start the server with |
Sorry I wanted to say "After killing one process the whole app is not responding" :) I use these versions:
There is obvious problem with killing uwsgi workers, because if I press Ctrl+C during running uwsgi I see timeout at the end of log (it takes more than 30 seconds).
There is no waiting with
|
One of my colleagues has the same problem on Mac |
Maybe I could create Dockerfile + main.py with this bug if you want :) |
Thank you for details! I can indeed reproduce the problem, the program freezes when I press Ctrl+C. I'm not sure what causes the problem exactly, but if you configure the if __name__ == "__main__":
logger.remove()
logger.add("file.log", enqueue=True)
logger.info("Logger is configured") |
It seems good but it's not. This code is not executed as expected. Then the behaviour is the same like default loguru settings (no logging to file). |
I believe I know where is problem. I created pull request #325. I hope it will helps you :) |
Hi again @dstlmrk! Quick question about the snippet you shared a few comments above: # main.py
import os
from loguru import logger
print("pid=%s, configure logger" % os.getpid())
logger.remove()
logger.add("file.log", enqueue=True)
logger.info("Logger is configured")
from flask import Flask
application = Flask(__name__) Any reason for importing |
Hello @Delgan. I spent a few hours but probably I found the problem. It's combination forking and locking the SimpleQueue. When you are pushing/putting messages to/from The solution is to don't call Read more here: https://pythonspeed.com/articles/python-multiprocessing/ You can try this code. It includes SImpleQueue and forking (because it does uwsgi). You have to find the moment when the lock is copied in an acquired state (the child is completely dead). import os
import time
import multiprocessing
from threading import Thread
from os import fork, getpid
def queued_writer():
while True:
try:
print(f"[PID={os.getpid()}] queue.get()")
message = queue.get()
except Exception:
print("exception")
continue
if message is None:
break
if message is True:
continue
print(f"[PID={os.getpid()}] write '{message}'")
queue = multiprocessing.SimpleQueue()
thread = Thread(target=queued_writer, daemon=True, name="writer")
thread.start()
queue.put(
"Lorem ipsum dolor sit amet, consectetuer adipiscing elit."
"Mauris tincidunt sem sed arcu. Fusce wisi. Aliquam erat volutpat."
"Praesent id justo in neque elementum ultrices."
"Morbi scelerisque luctus velit. Mauris metus."
"Suspendisse sagittis ultrices augue. Pellentesque sapien."
)
if fork():
while True:
print("parent")
time.sleep(1)
else:
while True:
print("child")
time.sleep(1) |
Thank you for taking the time to investigate, it's much more what I was asking for! However, I am somewhat surprised because the problem you describe was fixed a few months ago! This led to the creation of the The article you shared describes the difficulties of mixing multithreading with multiprocessing and why it's discouraged. It was definitely a problem with To be more specific about the snippet you just shared. By running it multiple times, it indeed happens that the child seems blocked. Here is the twist: it happens because of As I said, this is a problem experimented and fixed a few months ago: #231 (comment) Maybe all this is closely or remotely associated with the compatibility issues between |
Yeah but Btw the problem is not |
Yes, indeed. When I said that the problem came from |
Bumps [loguru](https://github.com/Delgan/loguru) from 0.5.0 to 0.5.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/Delgan/loguru/releases">loguru's releases</a>.</em></p> <blockquote> <h2>0.5.3</h2> <ul> <li>Fix child process possibly hanging at exit while combining <code>enqueue=True</code> with third party library like <code>uwsgi</code> (<a href="https://redirect.github.com/Delgan/loguru/issues/309">#309</a>, thanks <a href="https://github.com/dstlmrk"><code>@dstlmrk</code></a>).</li> <li>Fix possible exception during formatting of non-string messages (<a href="https://redirect.github.com/Delgan/loguru/issues/331">#331</a>).</li> </ul> <h2>0.5.2</h2> <ul> <li>Fix <code>AttributeError</code> within handlers using <code>serialize=True</code> when calling <code>logger.exception()</code> outside of the context of an exception (<a href="https://redirect.github.com/Delgan/loguru/issues/296">#296</a>).</li> <li>Fix error while logging an exception containing a non-picklable <code>value</code> to a handler with <code>enqueue=True</code> (<a href="https://redirect.github.com/Delgan/loguru/issues/298">#298</a>).</li> <li>Add support for async callable classes (with <code>__call__</code> method) used as sinks (<a href="https://redirect.github.com/Delgan/loguru/issues/294">#294</a>, thanks <a href="https://github.com/jessekrubin"><code>@jessekrubin</code></a>).</li> </ul> <h2>0.5.1</h2> <ul> <li>Modify the way the <code>extra</code> dict is used by <code>LogRecord</code> in order to prevent possible <code>KeyError</code> with standard <code>logging</code> handlers (<a href="https://redirect.github.com/Delgan/loguru/issues/271">#271</a>).</li> <li>Add a new <code>default</code> optional argument to <code>logger.catch()</code>, it should be the returned value by the decorated function in case an error occurred (<a href="https://redirect.github.com/Delgan/loguru/issues/272">#272</a>).</li> <li>Fix <code>ValueError</code> when using <code>serialize=True</code> in combination with <code>logger.catch()</code> or <code>logger.opt(record=True)</code> due to circular reference of the <code>record</code> dict (<a href="https://redirect.github.com/Delgan/loguru/issues/286">#286</a>).</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/Delgan/loguru/blob/master/CHANGELOG.rst">loguru's changelog</a>.</em></p> <blockquote> <h1><code>0.5.3</code>_ (2020-09-20)</h1> <ul> <li>Fix child process possibly hanging at exit while combining <code>enqueue=True</code> with third party library like <code>uwsgi</code> (<code>[#309](Delgan/loguru#309) <https://github.com/Delgan/loguru/issues/309></code><em>, thanks <code>@dstlmrk <https://github.com/dstlmrk></code></em>).</li> <li>Fix possible exception during formatting of non-string messages (<code>[#331](Delgan/loguru#331) <https://github.com/Delgan/loguru/issues/331></code>_).</li> </ul> <h1><code>0.5.2</code>_ (2020-09-06)</h1> <ul> <li>Fix <code>AttributeError</code> within handlers using <code>serialize=True</code> when calling <code>logger.exception()</code> outside of the context of an exception (<code>[#296](Delgan/loguru#296) <https://github.com/Delgan/loguru/issues/296></code>_).</li> <li>Fix error while logging an exception containing a non-picklable <code>value</code> to a handler with <code>enqueue=True</code> (<code>[#298](Delgan/loguru#298) <https://github.com/Delgan/loguru/issues/298></code>_).</li> <li>Add support for async callable classes (with <code>__call__</code> method) used as sinks (<code>[#294](Delgan/loguru#294) <https://github.com/Delgan/loguru/pull/294></code><em>, thanks <code>@jessekrubin <https://github.com/jessekrubin></code></em>).</li> </ul> <h1><code>0.5.1</code>_ (2020-06-12)</h1> <ul> <li>Modify the way the <code>extra</code> dict is used by <code>LogRecord</code> in order to prevent possible <code>KeyError</code> with standard <code>logging</code> handlers (<code>[#271](Delgan/loguru#271) <https://github.com/Delgan/loguru/issues/271></code>_).</li> <li>Add a new <code>default</code> optional argument to <code>logger.catch()</code>, it should be the returned value by the decorated function in case an error occurred (<code>[#272](Delgan/loguru#272) <https://github.com/Delgan/loguru/issues/272></code>_).</li> <li>Fix <code>ValueError</code> when using <code>serialize=True</code> in combination with <code>logger.catch()</code> or <code>logger.opt(record=True)</code> due to circular reference of the <code>record</code> dict (<code>[#286](Delgan/loguru#286) <https://github.com/Delgan/loguru/issues/286></code>_).</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/Delgan/loguru/commit/f31e97142adc1156693a26ecaf47208d3765a6e3"><code>f31e971</code></a> Bump version to 0.5.3</li> <li><a href="https://github.com/Delgan/loguru/commit/3394fdbd04efa7f5ec9c9625ec4ec34ca72b16e7"><code>3394fdb</code></a> Update Changelog to reference <a href="https://redirect.github.com/Delgan/loguru/issues/309">#309</a> fix</li> <li><a href="https://github.com/Delgan/loguru/commit/44f677172ccf10ec60cbc46e4ac77394429211a7"><code>44f6771</code></a> Fix removing simple queue in child processes (<a href="https://redirect.github.com/Delgan/loguru/issues/325">#325</a>)</li> <li><a href="https://github.com/Delgan/loguru/commit/296635c33cbc735c3367728f24f07545dbfe19c5"><code>296635c</code></a> Fix possible exception while formatting non-string message (<a href="https://redirect.github.com/Delgan/loguru/issues/331">#331</a>)</li> <li><a href="https://github.com/Delgan/loguru/commit/b77f4fd23ac380fa2521dd2039c961af1218d1d1"><code>b77f4fd</code></a> Correct simple typo in docs, propogates -> propagates (<a href="https://redirect.github.com/Delgan/loguru/issues/323">#323</a>)</li> <li><a href="https://github.com/Delgan/loguru/commit/e48f3e49228ada2f6495dd7eedc4ecd03530a1f6"><code>e48f3e4</code></a> Bump version to 0.5.2</li> <li><a href="https://github.com/Delgan/loguru/commit/d93c19f6c1c53cc14e9afcb28b770f8aa43c28cd"><code>d93c19f</code></a> Flesh out the few remaining incomplete annotations (<a href="https://redirect.github.com/Delgan/loguru/issues/316">#316</a>)</li> <li><a href="https://github.com/Delgan/loguru/commit/084dffd16211e8494419ed5cf9c6859de8e76123"><code>084dffd</code></a> Fix tests fails (since pytest upgrade) due to standard logging misuse</li> <li><a href="https://github.com/Delgan/loguru/commit/9a40e58c74e786149b6a4c6097e24d50e0b40459"><code>9a40e58</code></a> Fix error with "enqueue=True" and non-picklable exception (<a href="https://redirect.github.com/Delgan/loguru/issues/298">#298</a>)</li> <li><a href="https://github.com/Delgan/loguru/commit/6f86f487590fe8f264a3bb26c722bf4be354687b"><code>6f86f48</code></a> Refactor usage of "inspect" functions for simplification</li> <li>Additional commits viewable in <a href="https://github.com/Delgan/loguru/compare/0.5.0...0.5.3">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=loguru&package-manager=pip&previous-version=0.5.0&new-version=0.5.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/pinecone-io/pinecone-python-client/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Hello,
I use this definition for compressing but I expected different behaviour:
All my raw log files are compressed every night. It's working. But I don't need them (uncompressed files) if they are already compressed. Loguru doesn't remove them and I don't know if it's wanted behaviour. Because now it's not saving any space on disc.
Should I use my custom callable which removes all old uncompromised files?
The text was updated successfully, but these errors were encountered: