Configuration option to output logs in logfmt #1022

azuredream · 2024-09-12T02:19:50Z

Add ability to configure Redis to output logs in logfmt (See https://brandur.org/logfmt) as well as configure timestamp format options to more standard ISO 8601 or unix timestamp.

This change is implemented by two configs:
log-format: Either default or logfmt.
log-timestamp-format: default, iso8601, or unix.

#1006

$ ./valkey-server  /home/zhaoz12/git/valkey/valkey/valkey.conf
pid=109463 role=RDB/AOF timestamp="2024-09-10T20:37:25.738-04:00" level=warning message="WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect."
pid=109463 role=RDB/AOF timestamp="2024-09-10T20:37:25.738-04:00" level=notice message="oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo"
pid=109463 role=RDB/AOF timestamp="2024-09-10T20:37:25.738-04:00" level=notice message="Valkey version=255.255.255, bits=64, commit=affbea5d, modified=1, pid=109463, just started"
pid=109463 role=RDB/AOF timestamp="2024-09-10T20:37:25.738-04:00" level=notice message="Configuration loaded"
pid=109463 role=master timestamp="2024-09-10T20:37:25.738-04:00" level=notice message="monotonic clock: POSIX clock_gettime"
pid=109463 role=master timestamp="2024-09-10T20:37:25.739-04:00" level=warning message="Failed to write PID file: Permission denied"

Signed-off-by: azuredream <[email protected]>

codecov · 2024-09-12T02:44:33Z

Codecov Report

Attention: Patch coverage is 24.07407% with 41 lines in your changes missing coverage. Please review.

Project coverage is 70.65%. Comparing base (262d970) to head (ca0cedf).
Report is 18 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/server.c	24.07%	41 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1022      +/-   ##
============================================
+ Coverage     70.61%   70.65%   +0.03%     
============================================
  Files           114      114              
  Lines         61734    63125    +1391     
============================================
+ Hits          43595    44602    +1007     
- Misses        18139    18523     +384

Files with missing lines	Coverage Δ
src/config.c	`78.84% <ø> (+0.14%)`	⬆️
src/server.h	`100.00% <ø> (ø)`
src/server.c	`87.67% <24.07%> (-0.97%)`	⬇️

... and 89 files with indirect coverage changes

madolson · 2024-09-12T04:48:50Z

😀 thanks for opening this, we're working on launching the next version but will take a look shortly.

PingXie · 2024-09-12T05:49:46Z

I like this change. Free-form log lines can be tricky to parse and search, and this feels like a good step toward adopting more structured logging. We could even extend it further to include properties like "area" (e.g., "cluster," "normal replication," "dual channel replication," etc.). Thanks for your work on this, @azuredream!

Signed-off-by: azuredream <[email protected]>

src/server.c

src/server.h

src/server.c

madolson · 2024-10-06T18:17:44Z

@valkey-io/core-team Please indicate your vote on this comment. The open questions I still have are if we want to also do the larger investigation before 8.1 launches about also including the component the log was generated from. We also have the JSON format to add on.

zuiderkwast

I've reviewed this before in the open source redis time. I like it, just a few comments.

It seems that JSON is more popular. We can add that too in another PR.

Configurable time format is good.

src/server.h

src/server.c

madolson · 2024-10-07T15:12:03Z

src/config.c

+configEnum log_timestamp_format_enum[] = {{"default", LOG_TIMESTAMP_DEFAULT},
+ {"iso8601", LOG_TIMESTAMP_ISO8601},
+ {"unix", LOG_TIMESTAMP_UNIX},


Suggested change

configEnum log_timestamp_format_enum[] = {{"default", LOG_TIMESTAMP_DEFAULT},

{"iso8601", LOG_TIMESTAMP_ISO8601},

{"unix", LOG_TIMESTAMP_UNIX},

configEnum log_timestamp_format_enum[] = {{"legacy", LOG_TIMESTAMP_LEGACY},

{"iso8601", LOG_TIMESTAMP_ISO8601},

{"milliseconds", LOG_TIMESTAMP_MILLISECONDS},

@azuredream Was there a reason we added the unix milliseconds timestamp option? While discussing it, we weren't sure if it was needed.

(@madolson The PR currently emits it as seconds, not milliseconds, since the epoch.)

If we want the epoch, I think millisecond timestamp makes a lot more sense. A lot of important events happen sub-second.

madolson · 2024-10-07T18:43:29Z

src/server.c

+ snprintf(buf + off, sizeof(buf) - off, "%03d%s", (int)tv.tv_usec / 1000, tzbuf);
+ break;
+
+ case LOG_TIMESTAMP_UNIX: snprintf(buf, sizeof(buf), "%ld", tv.tv_sec); break;


I think this should be in milliseconds, not seconds.

Signed-off-by: azuredream <[email protected]>

azuredream · 2024-10-12T17:37:51Z

@madolson @zuiderkwast
PR updated as you suggested.
Thanks for reviewing.

Signed-off-by: azuredream <[email protected]>

azuredream · 2024-10-12T18:27:01Z

Demo logs:

log-format logfmt
log-timestamp-format milliseconds

pid=557888 role=RDB/AOF timestamp="1728757076.562" level=notice message="oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo"

log-format legacy
log-timestamp-format milliseconds

558253:C 1728757310.034 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo

log-format logfmt
log-timestamp-format iso8601

pid=558427 role=RDB/AOF timestamp="2024-10-12T14:24:28.141-04:00" level=notice message="oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo"

log-format legacy
log-timestamp-format iso8601

558523:C 2024-10-12T14:24:57.923-04:00 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo

log-format logfmt
log-timestamp-format legacy

pid=558599 role=RDB/AOF timestamp="12 Oct 2024 14:25:59.813" level=notice message="oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo"

log-format legacy
log-timestamp-format legacy

558689:C 12 Oct 2024 14:26:30.872 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo

Signed-off-by: azuredream <[email protected]>

src/server.c

Signed-off-by: azuredream <[email protected]>

azuredream · 2024-10-17T01:38:23Z

src/server.c

 /* Low level logging. To use only for very big messages, otherwise
 * serverLog() is to prefer. */
-void serverLogRaw(int level, const char *msg) {
+void serverLogRaw(int level, char *msg) {


Mutability modified.

Hm... I hope this works. I see many places in the code where this function is called with a string literal, like

serverLogRaw(LL_WARNING, "Failed applying new module configuration. Restoring previous settings.");

This can't be mutable, I think...

zuiderkwast · 2024-10-18T07:30:59Z

src/server.c

+ while (msg[src] != '\0') {
+ if (msg[src] == '"') {
+ msg[dst++] = '\'';
+ } else if (msg[src] != '\n' && msg[src] != '\r') {
+ msg[dst++] = msg[src];
+ }
+ src++;


Just an idea: If we replace \r and \n with a space instead, then we don't need to copy/move the rest of the string. We don't need the dst index.

Suggested change

while (msg[src] != '\0') {

if (msg[src] == '"') {

msg[dst++] = '\'';

} else if (msg[src] != '\n' && msg[src] != '\r') {

msg[dst++] = msg[src];

}

src++;

while (msg[src] != '\0') {

if (msg[src] == '"') {

msg[src] = '\'';

} else if (msg[src] == '\n' && msg[src] == '\r') {

msg[src] = ' ';

}

src++;

src/server.c

Signed-off-by: azuredream <[email protected]>

azuredream · 2024-10-19T00:48:06Z

Code updateded.

src/server.c

Signed-off-by: azuredream <[email protected]>

zuiderkwast

Looks good to me now.

One of the commits is missing sign-off. See the Details link on the DCO CI job. Can you fix it? You can squash and force-push as you want.

See also the clang-format CI job. It's a trailing whitespace on one line.

Is the Valkey logo logged on startup? It must look very weird with logfmt?

zuiderkwast · 2024-10-19T20:04:37Z

src/server.c

 } else {
- role_char = (server.primary_host ? 'S' : 'M'); /* replica or Primary. */
+ role_index = (server.primary_host ? 2 : 3); /* Slave or Master. */


Nit: We should avoid slave and master even in comments.

Suggested change

role_index = (server.primary_host ? 2 : 3); /* Slave or Master. */

role_index = (server.primary_host ? 2 : 3); /* replica or primary. */

Configuration option to output logs in logfmt

2253944

Signed-off-by: azuredream <[email protected]>

azuredream mentioned this pull request Sep 12, 2024

[NEW] Output logs as JSON #1006

Open

madolson added the major-decision-pending Major decision pending by TSC team label Sep 12, 2024

Format checker forced these changes

820111a

Signed-off-by: azuredream <[email protected]>

madolson reviewed Oct 6, 2024

View reviewed changes

zuiderkwast reviewed Oct 6, 2024

View reviewed changes

src/server.h Outdated Show resolved Hide resolved

src/server.h Outdated Show resolved Hide resolved

madolson reviewed Oct 7, 2024

View reviewed changes

src/server.c Outdated Show resolved Hide resolved

madolson reviewed Oct 7, 2024

View reviewed changes

madolson added major-decision-approved Major decision approved by TSC team and removed major-decision-pending Major decision pending by TSC team labels Oct 7, 2024

madolson reviewed Oct 7, 2024

View reviewed changes

azuredream added 3 commits October 12, 2024 00:07

Merge branch 'valkey-io:unstable' into i1006_logasjason

cd5edb6

Merge branch 'valkey-io:unstable' into i1006_logasjason

ee57283

Configuration option to output logs in logfmt

23e889c

Signed-off-by: azuredream <[email protected]>

Configuration option to output logs in logfmt

7ca8f9b

Signed-off-by: azuredream <[email protected]>

Format checker forced these changes

c494d5c

Signed-off-by: azuredream <[email protected]>

zuiderkwast reviewed Oct 14, 2024

View reviewed changes

src/server.c Outdated Show resolved Hide resolved

Configuration option to output logs in logfmt

5e4b307

zuiderkwast reviewed Oct 14, 2024

View reviewed changes

src/server.c Outdated Show resolved Hide resolved

madolson self-requested a review October 15, 2024 04:41

filter out invalid logfmt chars

299387d

Signed-off-by: azuredream <[email protected]>

azuredream commented Oct 17, 2024

View reviewed changes

zuiderkwast reviewed Oct 18, 2024

View reviewed changes

src/server.c Outdated Show resolved Hide resolved

Configuration option to output logs in logfmt

06e1ef4

Signed-off-by: azuredream <[email protected]>

zuiderkwast reviewed Oct 19, 2024

View reviewed changes

src/server.c Outdated Show resolved Hide resolved

Configuration option to output logs in logfmt

ca0cedf

Signed-off-by: azuredream <[email protected]>

zuiderkwast approved these changes Oct 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configuration option to output logs in logfmt #1022

Configuration option to output logs in logfmt #1022

azuredream commented Sep 12, 2024

codecov bot commented Sep 12, 2024 •

edited

Loading

madolson commented Sep 12, 2024

PingXie commented Sep 12, 2024

madolson commented Oct 6, 2024

zuiderkwast left a comment

madolson Oct 7, 2024 •

edited

Loading

zuiderkwast Oct 7, 2024

madolson Oct 7, 2024

madolson Oct 7, 2024 •

edited

Loading

azuredream commented Oct 12, 2024

azuredream commented Oct 12, 2024 •

edited

Loading

azuredream Oct 17, 2024

zuiderkwast Oct 18, 2024

zuiderkwast Oct 18, 2024

azuredream commented Oct 19, 2024

zuiderkwast left a comment •

edited

Loading

zuiderkwast Oct 19, 2024

	role_index = (server.primary_host ? 2 : 3); /* Slave or Master. */
	role_index = (server.primary_host ? 2 : 3); /* replica or primary. */

Configuration option to output logs in logfmt #1022

Are you sure you want to change the base?

Configuration option to output logs in logfmt #1022

Conversation

azuredream commented Sep 12, 2024

codecov bot commented Sep 12, 2024 • edited Loading

Codecov Report

madolson commented Sep 12, 2024

PingXie commented Sep 12, 2024

madolson commented Oct 6, 2024

zuiderkwast left a comment

Choose a reason for hiding this comment

madolson Oct 7, 2024 • edited Loading

Choose a reason for hiding this comment

zuiderkwast Oct 7, 2024

Choose a reason for hiding this comment

madolson Oct 7, 2024

Choose a reason for hiding this comment

madolson Oct 7, 2024 • edited Loading

Choose a reason for hiding this comment

azuredream commented Oct 12, 2024

azuredream commented Oct 12, 2024 • edited Loading

azuredream Oct 17, 2024

Choose a reason for hiding this comment

zuiderkwast Oct 18, 2024

Choose a reason for hiding this comment

zuiderkwast Oct 18, 2024

Choose a reason for hiding this comment

azuredream commented Oct 19, 2024

zuiderkwast left a comment • edited Loading

Choose a reason for hiding this comment

zuiderkwast Oct 19, 2024

Choose a reason for hiding this comment

codecov bot commented Sep 12, 2024 •

edited

Loading

madolson Oct 7, 2024 •

edited

Loading

madolson Oct 7, 2024 •

edited

Loading

azuredream commented Oct 12, 2024 •

edited

Loading

zuiderkwast left a comment •

edited

Loading