Skip to content

Noobaa Support Cases Help Guide for ODF

Jump to bottom
shirady edited this page Sep 9, 2024 · 11 revisions

Support Docs Links

  • Noobaa performance tuning
    In many cases, the default resource limits\requests allocated for noobaa pods in ODF are not sufficient for the actual use of noobaa. You can check noobaa CR to see if the defaults have changed (e.g., the default for the DB is 0.5 CPU) and recommend increasing it if not. many times, this is not the root cause of the issue, but it can help in further diagnostic of the system by eliminating other issues.
  • How to rebuild Noobaa in ODF
    This should be used carefully since it will result in a new empty system and cause loss of all existing data in Noobaa. It is useful when the customer is not using Noobaa, but it interferes with the ODF use (e.g.: upgrade issues, irrelevant alerts, etc.).
    In some cases, this can be used as a last resort when there is no other way to restore the system. in that case, if there is a backup of noobaa it is possible to try and restore from backup (see backup and restore support case).
  • Backup and Restore

Debugging Procedures

  • Checking whether a proxy is configured can reveal and explain all sorts of network problems - from certificates to request timeouts. It's possible to check whether a proxy is configured by opening a shell on the core and operator pods, running env, and looking for an environment variables called HTTP_PROXY, HTTPS_PROXY, or NO_PROXY

  • Postgres logs location in the DB pod - /var/lib/pgsql/data/userdata/log

  • Debugging Postgres CPU issues
    When the DB is maxing its CPU allocation, you can try using Postgres pg_stat_statements extension to check which queries take the most time to run. By also enabling query logging, you can identify the exact code that produces this query

  • Setting debug level

    • Dynamically without pods restarts (recommended) To get core and endpoints logs higher than L0, you can increase the debug level. It's important to reduce the level back to 0 after the logs were taken to avoid performance issues due to logging. Also, notice that the debug level is not persistent and is set back to 0 after a pod restart. In ODF 4.13 and up there is a CLI command: 
      noobaa system set-debug-level 5
      On versions 4.12 and older it can be done using an API command 
      noobaa api redirector publish_to_cluster '{ "target": "" , "method_api": "debug_api", "method_name": "set_debug_level", "request_params": {"level": 5, "module": "core"} }'
    • Statically using noobaa-config in the config map noobaa-config, set "NOOBAA_LOG_LEVEL: all". This will cause all pods to restart with a high debug level. It will also be persistent until it is restored to "NOOBAA_LOG_LEVEL: default_level"

Other

Activity Logs

In case you want to review the events that occurs in the system (and in some cases you want since the activity is not covered in the logs).
Steps:

  1. Connect to the DB pod: kubectl exec -it noobaa-db-pg-0 -n <your-rnamespace> -- bash
  2. Type: psql
  3. Move the relevant user: \c nbcore
  4. To see all the tables in the system: \dt
  5. Run the command to easily see the logs: SELECT jsonb_pretty(data) FROM activitylogs;

Notes:

  • You can filter the queries by timestamp, actor, etc.
  • Under the hood the logs are created using the Dispatcher.instance().activity function in the code.

Environment variables

If you want to add environment variables in a cluster that were defined in the config.js, you can override the value with the prefix CONFIG_JS_ in the variable name, for example:

If there was a variable SOME_VAR with value 1 in the config.js but not exported in the core, and you wish to update its value only in noobaa-core.

  1. kubectl exec statefulset/noobaa-core -c core -- printenv | grep SOME_VAR (should be empty).
  2. kubectl set env statefulset/noobaa-core -c core CONFIG_JS_SOME_VAR=3 (a different number that we defined in the config, then it should output statefulset.apps/noobaa-core env updated and noobaa-core-0 pod will be terminated and start running again).
  3. kubectl exec statefulset/noobaa-core -c core -- printenv | grep SOME_VAR (should be SOME_VAR=3).

Check connection (since 5.14)

If you wish to check the connection from backingstores or namesapcestores to the cloud provider and use the same credentials, you can use the commands (using the noobaa CLI):

  • For one backingstore: noobaa diagnostics analyze backingstore <name>.
  • For one namespacestore: noobaa diagnostics analyze namespacestore <name>.
  • For all backingstores and namespacestores: noobaa diagnostics analyze resources.
Clone this wiki locally