DAOS-16327 control: Update dmg storage query usage for MD-on-SSD P2 #15418

tanabarr · 2024-10-28T23:47:30Z

Add --mem-ratio and --show-usable flags to dmg storage query usage to
improve usability in MD-on-SSD phase-2 mode. Update table output to
show space usage per rank and tier in MD-on-SSD mode. Add relevant
pool create and storage query usage cmd examples in the admin guide.

Required-githooks: true

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

github-actions · 2024-10-28T23:47:51Z

Ticket title is 'Update dmg storage query usage for MD-on-SSD P2'
Status is 'In Review'
Labels: 'control-plane,usability,md_on_ssd2'
https://daosio.atlassian.net/browse/DAOS-16327

knard-intel

Some minor fix needed.

docs/admin/pool_operations.md

src/control/cmd/dmg/pretty/storage_query.go

src/control/server/storage/bdev.go

daosbuild1 · 2024-11-02T03:54:38Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15418/4/display/redirect

daosbuild1 · 2024-11-04T13:33:56Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15418/5/testReport/

daltonbohning · 2024-11-04T16:44:01Z

Does this need a rebase before reviews?

brianjmurrell · 2024-11-04T16:47:34Z

Does this need a rebase before reviews?

It definitely needs a rebase (merge at this point since reviewing has started) so probably better to do that before more reviewing happens.

brianjmurrell

Packaging changes LGTM.

Features: control Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-large-md-on-ssd: false Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

knard-intel

Overall is good for me. just minor fixes which could be done in a follow-up PR.

knard-intel · 2024-11-05T14:04:22Z

docs/admin/pool_operations.md


-$ dmg pool create bob2 --size 100% --mem-ratio 50%
+3. If we then try the same with 2 ranks/engines where bdev


As for the first example, there is only one rank from my understanding.

knard-intel · 2024-11-05T14:07:34Z

docs/admin/pool_operations.md

+
+6. Adding another engine/rank on the same host results in more than double DATA
+capacity because RAM-disk capacity is halved across two engines/ranks on the same
+host and this results in a reduction of META and increase in DATA per-rank sizes.


NIT: an explanation on why total Memory File Size (i.e. sum of the two ranks) is lower than the one engine configuration, could be helpful.

knard-intel · 2024-11-05T14:11:13Z

docs/admin/pool_operations.md

+---- -----
+T1   data,meta,wal
+
+Rank T1-Total T1-Usable T1-Usage


Explanation of the last columns meaning could be helpful. I am not sure if it is the calculated according to the available size or the usable one

daosbuild1 · 2024-11-05T22:55:20Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15418/7/testReport/

src/control/server/storage/bdev.go

src/control/lib/control/storage.go

docs/admin/pool_operations.md

kjacque · 2024-11-06T01:09:15Z

src/control/cmd/dmg/storage_query.go

+		pretty.PrintHostStorageUsageMap(resp.HostStorage, &out)
+	}
+	if dbg.Len() > 0 {
+		cmd.Debugf("%s", dbg.String())


Suggested change

cmd.Debugf("%s", dbg.String())

cmd.Debug(dbg.String())

this is useful because some parsing of escape characters is different if you run it through format

Ah, the newlines aren't parsed when you just print the string? Didn't know that. That sort of seems like a bug in our logging to be honest.

actually I think it was something to do with the percentage sign parsing

Oh, hm. That makes some sense, but it seems like the non-format version shouldn't be mangling anything at all.

src/control/cmd/dmg/storage_query.go

src/control/cmd/dmg/storage_query_test.go

src/control/cmd/dmg/pretty/storage_query.go

Features: control Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-large-md-on-ssd: false Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

src/control/cmd/dmg/pretty/storage_query_test.go

tanabarr · 2024-11-08T21:33:15Z

bump @mjmac @knard-intel reviews please

knard-intel

LGTM

tanabarr requested review from a team as code owners October 28, 2024 23:47

tanabarr requested review from mjmac, kjacque and knard-intel October 28, 2024 23:48

knard-intel reviewed Oct 29, 2024

View reviewed changes

tanabarr force-pushed the tanabarr/control-queryusage-mdonssd-2 branch from 0644365 to ef771e6 Compare November 1, 2024 20:34

tanabarr requested review from a team as code owners November 1, 2024 20:34

tanabarr changed the base branch from feature/vos_on_blob_p2 to niu/vos_on_blob_p2/p2_landing November 1, 2024 20:36

tanabarr removed request for a team November 1, 2024 20:43

Base automatically changed from niu/vos_on_blob_p2/p2_landing to master November 4, 2024 13:02

gnailzenh requested review from a team as code owners November 4, 2024 13:02

brianjmurrell reviewed Nov 4, 2024

View reviewed changes

DAOS-16327 control: Update dmg storage query usage for MD-on-SSD P2

ece44f2

Features: control Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-large-md-on-ssd: false Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

tanabarr force-pushed the tanabarr/control-queryusage-mdonssd-2 branch from 78e0927 to ece44f2 Compare November 5, 2024 13:52

tanabarr requested a review from knard-intel November 5, 2024 13:54

knard-intel previously approved these changes Nov 5, 2024

View reviewed changes

kjacque reviewed Nov 6, 2024

View reviewed changes

address review comments from knard-intel and kjacque

2c7164e

Features: control Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-large-md-on-ssd: false Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

tanabarr dismissed knard-intel’s stale review via 2c7164e November 6, 2024 11:48

tanabarr requested review from kjacque and knard-intel November 6, 2024 11:49

kjacque reviewed Nov 8, 2024

View reviewed changes

src/control/cmd/dmg/pretty/storage_query_test.go Show resolved Hide resolved

tanabarr requested a review from kjacque November 8, 2024 09:07

kjacque approved these changes Nov 8, 2024

View reviewed changes

knard-intel approved these changes Nov 9, 2024

View reviewed changes

tanabarr self-assigned this Nov 11, 2024

tanabarr added control-plane work on the management infrastructure of the DAOS Control Plane meta-on-ssd Metadata on SSD Feature labels Nov 11, 2024

tanabarr requested a review from a team November 11, 2024 10:17

NiuYawei merged commit d4070e8 into master Nov 11, 2024
62 checks passed

NiuYawei deleted the tanabarr/control-queryusage-mdonssd-2 branch November 11, 2024 14:47

mjmac mentioned this pull request Nov 13, 2024

mjmac/DAOS 16787 google 2.6 #15498

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-16327 control: Update dmg storage query usage for MD-on-SSD P2 #15418

DAOS-16327 control: Update dmg storage query usage for MD-on-SSD P2 #15418

tanabarr commented Oct 28, 2024 •

edited

Loading

github-actions bot commented Oct 28, 2024 •

edited

Loading

knard-intel left a comment

daosbuild1 commented Nov 2, 2024

daosbuild1 commented Nov 4, 2024

daltonbohning commented Nov 4, 2024

brianjmurrell commented Nov 4, 2024

brianjmurrell left a comment

knard-intel left a comment

knard-intel Nov 5, 2024

tanabarr Nov 5, 2024

knard-intel Nov 5, 2024

tanabarr Nov 5, 2024

knard-intel Nov 5, 2024

daosbuild1 commented Nov 5, 2024

kjacque Nov 6, 2024

tanabarr Nov 6, 2024

kjacque Nov 8, 2024

tanabarr Nov 8, 2024

kjacque Nov 8, 2024

tanabarr commented Nov 8, 2024

knard-intel left a comment


		$ dmg pool create bob2 --size 100% --mem-ratio 50%
		3. If we then try the same with 2 ranks/engines where bdev

DAOS-16327 control: Update dmg storage query usage for MD-on-SSD P2 #15418

DAOS-16327 control: Update dmg storage query usage for MD-on-SSD P2 #15418

Conversation

tanabarr commented Oct 28, 2024 • edited Loading

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Oct 28, 2024 • edited Loading

knard-intel left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 2, 2024

daosbuild1 commented Nov 4, 2024

daltonbohning commented Nov 4, 2024

brianjmurrell commented Nov 4, 2024

brianjmurrell left a comment

Choose a reason for hiding this comment

knard-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tanabarr commented Nov 8, 2024

knard-intel left a comment

Choose a reason for hiding this comment

tanabarr commented Oct 28, 2024 •

edited

Loading

github-actions bot commented Oct 28, 2024 •

edited

Loading