New feature: support memory row CE threshold policy #150

zhuofeng6 · 2024-03-04T13:07:12Z

Introduction: Identify memory row faults in memory CE faults and isolate the physical memory pages where row faults occur. This method can effectively prevent CE storms or memory UCE faults caused by memory row failures.
Implementation: The system counts the number of CE faults in the same memory row within a specified period. If the number of CE faults exceeds the configured threshold, the system considers that the memory row may fail and isolates all physical pages recorded in the memory row.

Notes:

This function is disabled by default. You can enable it by configuring the'ROW_CE_ACTION' field in the '/etc/sysconfig/rasdaemon' configuration file.
If both row isolation and page isolation are enabled, page isolation is automatically disabled by default.
If the number of fault times in the DIMM CE fault information received by the rasdaemon is 0, the BIOS does not correctly parse the number of fault times when parsing the fault information. When a fault occurs, the rasdaemon process considers that the number of faults is 1 by default, which is the same as the kernel process.

mchehab · 2024-06-11T09:33:07Z

Please add a Signed-off-by for the patches you submit.

(please see: https://www.kernel.org/doc/html/latest/process/submitting-patches.html#developer-s-certificate-of-origin-1-1)

mchehab

If both row isolation and page isolation are enabled, page isolation is automatically
disabled by default.

I would be expecting to have a warning for users to know that raw isolation disabled page isolation. Also, I don't see what part of the code actually checks it.

mchehab · 2024-06-11T09:38:57Z

ras-page-isolation.h

+ unsigned long count;
+};
+
+


Nitpick: just one blank line.

mchehab · 2024-06-11T09:39:26Z

ras-page-isolation.h

+ int count;
+ time_t start;
+};
+#define ROW_LOCATION_FIELDS_NUM (DSM_FIELD_NUM > APEI_FIELD_NUM ? DSM_FIELD_NUM : APEI_FIELD_NUM)


Nitpick: please add a blank line after struct definitions.

zhuofeng6 · 2024-06-18T12:56:21Z

ras-page-isolation.c

@@ -102,6 +121,11 @@ static void page_offline_init(void)
 offline = OFFLINE_ACCOUNT;
 }

+ if (row_offline_action != OFFLINE_OFF) {
+ log(TERM, LOG_INFO, "row threshold is open, so turn off page threshold\n");


If both row isolation and page isolation are enabled, page isolation is automatically
disabled by default.

I would be expecting to have a warning for users to know that raw isolation disabled page isolation. Also, I don't see what part of the code actually checks it.

I checked here.

zhuofeng6 · 2024-06-19T01:12:26Z

Please add a Signed-off-by for the patches you submit.

(please see: https://www.kernel.org/doc/html/latest/process/submitting-patches.html#developer-s-certificate-of-origin-1-1)

Already added

mchehab · 2024-07-16T06:04:09Z

Please rebase on the top of upstream, and check compilation/warnings.

At least here, it is not building with gcc 14.1.1:

In file included from ras-page-isolation.c:24:
ras-page-isolation.h:100:48: warning: comparison between 'enum dsm_location_field_index' and 'enum apei_location_field_index' [-Wenum-compare]
  100 | #define ROW_LOCATION_FIELDS_NUM (DSM_FIELD_NUM > APEI_FIELD_NUM ? DSM_FIELD_NUM : APEI_FIELD_NUM)
      |                                                ^
ras-page-isolation.h:105:49: note: in expansion of macro 'ROW_LOCATION_FIELDS_NUM'
  105 |         int                     location_fields[ROW_LOCATION_FIELDS_NUM];
      |                                                 ^~~~~~~~~~~~~~~~~~~~~~~
ras-page-isolation.c: In function 'row_isolation_init':
ras-page-isolation.c:276:9: error: too few arguments to function 'parse_env_string'
  276 |         parse_env_string(&row_threshold, threshold_string);
      |         ^~~~~~~~~~~~~~~~
ras-page-isolation.c:198:13: note: declared here
  198 | static void parse_env_string(struct isolation *config, char *str, unsigned int size)
      |             ^~~~~~~~~~~~~~~~
ras-page-isolation.c:277:9: error: too few arguments to function 'parse_env_string'
  277 |         parse_env_string(&row_cycle, cycle_string);
      |         ^~~~~~~~~~~~~~~~
ras-page-isolation.c:198:13: note: declared here
  198 | static void parse_env_string(struct isolation *config, char *str, unsigned int size)
      |             ^~~~~~~~~~~~~~~~
ras-page-isolation.c: In function 'row_record_copy':
ras-page-isolation.h:100:48: warning: comparison between 'enum dsm_location_field_index' and 'enum apei_location_field_index' [-Wenum-compare]
  100 | #define ROW_LOCATION_FIELDS_NUM (DSM_FIELD_NUM > APEI_FIELD_NUM ? DSM_FIELD_NUM : APEI_FIELD_NUM)
      |                                                ^
ras-page-isolation.c:501:29: note: in expansion of macro 'ROW_LOCATION_FIELDS_NUM'
  501 |         for (int i = 0; i < ROW_LOCATION_FIELDS_NUM; i++) {
      |                             ^~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [Makefile:1256: rasdaemon-ras-page-isolation.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from ras-mc-handler.c:29:
ras-page-isolation.h:100:48: warning: comparison between 'enum dsm_location_field_index' and 'enum apei_location_field_index' [-Wenum-compare]
  100 | #define ROW_LOCATION_FIELDS_NUM (DSM_FIELD_NUM > APEI_FIELD_NUM ? DSM_FIELD_NUM : APEI_FIELD_NUM)
      |                                                ^
ras-page-isolation.h:105:49: note: in expansion of macro 'ROW_LOCATION_FIELDS_NUM'
  105 |         int                     location_fields[ROW_LOCATION_FIELDS_NUM];
      |                                                 ^~~~~~~~~~~~~~~~~~~~~~~
In file included from ras-events.c:45:
ras-page-isolation.h:100:48: warning: comparison between 'enum dsm_location_field_index' and 'enum apei_location_field_index' [-Wenum-compare]
  100 | #define ROW_LOCATION_FIELDS_NUM (DSM_FIELD_NUM > APEI_FIELD_NUM ? DSM_FIELD_NUM : APEI_FIELD_NUM)
      |                                                ^
ras-page-isolation.h:105:49: note: in expansion of macro 'ROW_LOCATION_FIELDS_NUM'
  105 |         int                     location_fields[ROW_LOCATION_FIELDS_NUM];
      |                                                 ^~~~~~~~~~~~~~~~~~~~~~~

- Introduction: Identify memory row faults in memory CE faults and isolate the physical memory pages where row faults occur. This method can effectively prevent CE storms or memory UCE faults caused by memory row failures. - Implementation: The system counts the number of CE faults in the same memory row within a specified period. If the number of CE faults exceeds the configured threshold, the system considers that the memory row may fail and isolates all physical pages recorded in the memory row. Notes: 1. This function is disabled by default. You can enable it by configuring the'ROW_CE_ACTION' field in the '/etc/sysconfig/rasdaemon' configuration file. 2. If both row isolation and page isolation are enabled, page isolation is automatically disabled by default. 3. If the number of fault times in the DIMM CE fault information received by the rasdaemon is 0, the BIOS does not correctly parse the number of fault times when parsing the fault information. When a fault occurs, the rasdaemon process considers that the number of faults is 1 by default, which is the same as the kernel process. Signed-off-by: zhuofeng <[email protected]>

zhuofeng6 · 2024-07-29T01:14:09Z

Please rebase on the top of upstream, and check compilation/warnings.

At least here, it is not building with gcc 14.1.1:

In file included from ras-page-isolation.c:24:
ras-page-isolation.h:100:48: warning: comparison between 'enum dsm_location_field_index' and 'enum apei_location_field_index' [-Wenum-compare]
  100 | #define ROW_LOCATION_FIELDS_NUM (DSM_FIELD_NUM > APEI_FIELD_NUM ? DSM_FIELD_NUM : APEI_FIELD_NUM)
      |                                                ^
ras-page-isolation.h:105:49: note: in expansion of macro 'ROW_LOCATION_FIELDS_NUM'
  105 |         int                     location_fields[ROW_LOCATION_FIELDS_NUM];
      |                                                 ^~~~~~~~~~~~~~~~~~~~~~~
ras-page-isolation.c: In function 'row_isolation_init':
ras-page-isolation.c:276:9: error: too few arguments to function 'parse_env_string'
  276 |         parse_env_string(&row_threshold, threshold_string);
      |         ^~~~~~~~~~~~~~~~
ras-page-isolation.c:198:13: note: declared here
  198 | static void parse_env_string(struct isolation *config, char *str, unsigned int size)
      |             ^~~~~~~~~~~~~~~~
ras-page-isolation.c:277:9: error: too few arguments to function 'parse_env_string'
  277 |         parse_env_string(&row_cycle, cycle_string);
      |         ^~~~~~~~~~~~~~~~
ras-page-isolation.c:198:13: note: declared here
  198 | static void parse_env_string(struct isolation *config, char *str, unsigned int size)
      |             ^~~~~~~~~~~~~~~~
ras-page-isolation.c: In function 'row_record_copy':
ras-page-isolation.h:100:48: warning: comparison between 'enum dsm_location_field_index' and 'enum apei_location_field_index' [-Wenum-compare]
  100 | #define ROW_LOCATION_FIELDS_NUM (DSM_FIELD_NUM > APEI_FIELD_NUM ? DSM_FIELD_NUM : APEI_FIELD_NUM)
      |                                                ^
ras-page-isolation.c:501:29: note: in expansion of macro 'ROW_LOCATION_FIELDS_NUM'
  501 |         for (int i = 0; i < ROW_LOCATION_FIELDS_NUM; i++) {
      |                             ^~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [Makefile:1256: rasdaemon-ras-page-isolation.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from ras-mc-handler.c:29:
ras-page-isolation.h:100:48: warning: comparison between 'enum dsm_location_field_index' and 'enum apei_location_field_index' [-Wenum-compare]
  100 | #define ROW_LOCATION_FIELDS_NUM (DSM_FIELD_NUM > APEI_FIELD_NUM ? DSM_FIELD_NUM : APEI_FIELD_NUM)
      |                                                ^
ras-page-isolation.h:105:49: note: in expansion of macro 'ROW_LOCATION_FIELDS_NUM'
  105 |         int                     location_fields[ROW_LOCATION_FIELDS_NUM];
      |                                                 ^~~~~~~~~~~~~~~~~~~~~~~
In file included from ras-events.c:45:
ras-page-isolation.h:100:48: warning: comparison between 'enum dsm_location_field_index' and 'enum apei_location_field_index' [-Wenum-compare]
  100 | #define ROW_LOCATION_FIELDS_NUM (DSM_FIELD_NUM > APEI_FIELD_NUM ? DSM_FIELD_NUM : APEI_FIELD_NUM)
      |                                                ^
ras-page-isolation.h:105:49: note: in expansion of macro 'ROW_LOCATION_FIELDS_NUM'
  105 |         int                     location_fields[ROW_LOCATION_FIELDS_NUM];
      |                                                 ^~~~~~~~~~~~~~~~~~~~~~~

Modified

mchehab requested changes Jun 11, 2024

View reviewed changes

zhuofeng6 force-pushed the ce_row branch from 292948c to 7ebc551 Compare June 18, 2024 12:53

zhuofeng6 commented Jun 18, 2024

View reviewed changes

zhuofeng6 force-pushed the ce_row branch 4 times, most recently from 809760a to 92fa716 Compare July 28, 2024 03:45

zhuofeng6 force-pushed the ce_row branch from 92fa716 to 90624a2 Compare July 28, 2024 03:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New feature: support memory row CE threshold policy #150

New feature: support memory row CE threshold policy #150

zhuofeng6 commented Mar 4, 2024

mchehab commented Jun 11, 2024

mchehab left a comment

mchehab Jun 11, 2024

zhuofeng6 Jun 18, 2024

mchehab Jun 11, 2024

zhuofeng6 Jun 18, 2024

zhuofeng6 Jun 18, 2024

zhuofeng6 commented Jun 19, 2024

mchehab commented Jul 16, 2024 •

edited

Loading

zhuofeng6 commented Jul 29, 2024

New feature: support memory row CE threshold policy #150

Are you sure you want to change the base?

New feature: support memory row CE threshold policy #150

Conversation

zhuofeng6 commented Mar 4, 2024

mchehab commented Jun 11, 2024

mchehab left a comment

Choose a reason for hiding this comment

mchehab Jun 11, 2024

Choose a reason for hiding this comment

zhuofeng6 Jun 18, 2024

Choose a reason for hiding this comment

mchehab Jun 11, 2024

Choose a reason for hiding this comment

zhuofeng6 Jun 18, 2024

Choose a reason for hiding this comment

zhuofeng6 Jun 18, 2024

Choose a reason for hiding this comment

zhuofeng6 commented Jun 19, 2024

mchehab commented Jul 16, 2024 • edited Loading

zhuofeng6 commented Jul 29, 2024

mchehab commented Jul 16, 2024 •

edited

Loading