log rotation ... controlling the name of the rotated file

[jakane]

I would like log rotation to work similar to a common pattern used in logrotate(8).

ie: current logs going to file.log
rotated files going to file-YYYYmmdd-HHMM.log

What I wish to do is control the {time} portion of the rotated file name. I would like to have something like '-{time:YYYYMMDD-HHMM!UTC}' instead of what looks like '.YYYY-MM-DD_HH-MM-SS_SSSSSS}' (where it is in local time).

How would I go about this? Otherwise the rotation, compression, retention works as I would expect.

Thanks

[Delgan]

Hi.

You can use a custom formatter by specifying the "{time}" in the filename, like this:

logger.add("file-{time:YYYYMMDD-HHMM!UTC}.log", rotation="5 MB")

Only downside is that if you stop and restart your application, it will create a new file instead of continuing to write to the previous one until rotation threshold is reached. If that's a problem, I think you can use the compression argument to rename the rotated file to a pattern that suits your needs.

[ns-kyuan]

Hi @Delgan ,
I have a similar problem. My handler is like this,

logger.add(
  sink='task-scheduler-bootstrap-{time:YYYY-MM}.log',
  format=LOG_FORMAT,
  rotation="monthly",
  compression="xz",
  level="INFO")

and my program encountered the following exception when entering March

Traceback (most recent call last):
  File "/home/nsiq/.local/lib/python3.8/site-packages/loguru/_ctime_functions.py", line 32, in get_ctime                                                                                                                                                                                                                        return float(os.getxattr(filepath, b"user.loguru_crtime"))
FileNotFoundError: [Errno 2] No such file or directory: 'task-scheduler-bootstrap-2020-02.log'

It seems that Feb's log was properly rotated out and compressed while the program still wanted to write to it. Is there a way to fix this?

Thank you!

[Delgan]

Hi @ns-kyuan. I opened a new issue so we can properly discuss your problem there: #220 😉

[Delgan]

@jakane Are you ok with the snippet I shared to modify the name of the rotated file?

[bybatkhuu]

@Delgan,
Does it really work?
Mine was like:
logger.add("app.out{time:YYYY-MM-DD}.log", rotation='01:00')
And it immediately creates a log file named: "app.out2021-05-12.log"
Should it be: "app.out.log"

And whenever it rotates: -> "app.out2021-05-12.2021-05-12_01-00-00_002130.log"
What I expected on rotation: "app.out2021-05-12.log"

loguru version: 0.5.3

Am I doing wrong, or is there any good code snippet to handle this problem?
I couldn't find any solution in the documentation.

I really desperately need this...

[Delgan]

@bybatkhuu Yes, if you use the {time} formative directive, then it's expected to be part of the filename created (the date at the moment the handler starts is used).

You're seeing file named "app.out2021-05-12.2021-05-12_01-00-00_002130.log" probably because otherwise the files before and after rotation would have had the same names. So Loguru renamed the old file by post-fixing the complete date. Here is what I guess happened:

1. Application is started at 00:30 and the file "app.out2021-05-12.log" is created
2. Rotation occurs at 01:00: Loguru computes the new filename which is "app.out2021-05-12.log"
3. Loguru realizes there already exists a file with such name, so it renames it to avoid conflict
4. The rotated file is created with the name "app.out2021-05-12.log"

This behavior makes sense if you use logger.add("app.out.log", rotation="01:00") for example, as it automatically creates one new file per rotation.

In your case, if your application was started the 2021-05-11 and ran continuously without being restarted, then you would have had two files named "app.out2021-05-11.log" and "app.out2021-05-12.log" (because at the time of rotation, the names of the files were already distinct and did not need to be renamed).

I think you can achieve what you're looking for by using a custom compression function in charge of renaming your rotated file as you wish.

import os
from datetime import datetime

def rename_rotated_file(filepath):
    now = datetime.now()
    new_path = os.path.dirname(filepath) + "/app.out{:%Y-%m-%d}.log".format(now)
    os.rename(filepath, new_path)

logger.add("app.out.log", rotation="01:00", compression=rename_rotated_file)

[bybatkhuu]

@Delgan, Thank you.
Yeah, it works on one file sink. But if there are many file sinks, it doesn't work well.
Anyway, I managed to handle this by modifying loguru source code (a little bit hardcode.).
BTW, great work! @Delgan :D

[Delgan]

@bybatkhuu Feel free to open a new issue if you want to investigate this further. 😉

Maybe there is a way to avoid modifying Loguru's source code, and thus to benefit easily from the next updates.

[mandarvaze]

@Delgan I wonder if it is possible to (optionally) override generate_rename_path rather than via compression

[Delgan]

@mandarvaze I'm not really planning to expose this function to the public API. What is bothering you with the compression workaround?

[mandarvaze]

@Delgan When I debugged, I reached generate_rename_path which is what is used currently. So I suggested that, is all.
I'm OK with compression work around.

[mancioshell]

Hi @Delgan , i would like to customize rotate file name with custom compression function as you have suggested.
But i'm in trouble becuase i'm currently rotating file based on date (with format: yyyy-MM-dd_HH) and size. So in my case it could be possible to have 2 rotate files with same name (in case rotation is made by size at the same hour).
Is it possible to access the index of rotated file in case of size rotation ?

[Delgan]

@mancioshell I'm not sure if I fully understand your problem, but I don't think it's possible. Can you check that os.path.exists() is False before renaming the file?

[mancioshell]

@Delgan I'll try to explain my problem with more details.

My goal is to rotate the file in 2 cases (or if it exceeds a particular size e.g. 100 MB, or every hour).

My rotated filename should be : /var/log/platform/{yyyy-MM-dd_HH}.app.sdk-log.zip

If an application exceeds the 100 MB size in the same hour, it could override the existing zip file. So i need an index to track the already existing rotated files, so i could write to a file with the following pattern:
/var/log/platform/{yyyy-MM-dd_HH}.app.sdk-log.%i.zip

where %i is the index

Is it possible ?

[Delgan]

@mancioshell Thanks for the example.

There is no such index available, you need to enumerate the existing files until you find an available index, or increment an index from a variable.