DAOS-15971 dfs: add support for mounting a snapshot #15321

johannlombardi · 2024-10-15T13:50:05Z

Add new dfs interface to mount a specific snapshot.
Provide dfuse command line options to use this feature.

Signed-off-by: Johann Lombardi <[email protected]>

Add new dfs interface to mount a specific snapshot. Provide dfuse command line options to use this feature. Signed-off-by: Johann Lombardi <[email protected]>

Signed-off-by: Johann Lombardi <[email protected]>

github-actions · 2024-10-15T13:50:24Z

Ticket title is 'Add ability to mount a snapshot via dfuse'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-15971

daosbuild1 · 2024-10-15T13:56:06Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/1/execution/node/383/log

daosbuild1 · 2024-10-15T13:56:17Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/1/execution/node/387/log

daosbuild1 · 2024-10-15T13:56:45Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/1/execution/node/334/log

daosbuild1 · 2024-10-15T13:57:22Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/1/execution/node/357/log

Signed-off-by: Johann Lombardi <[email protected]>

daosbuild1 · 2024-10-15T13:58:20Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/1/execution/node/328/log

daosbuild1 · 2024-10-15T14:01:24Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/2/execution/node/388/log

daosbuild1 · 2024-10-15T14:01:34Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/2/execution/node/386/log

daosbuild1 · 2024-10-15T14:02:13Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/2/execution/node/370/log

daosbuild1 · 2024-10-15T14:02:37Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/2/execution/node/360/log

daosbuild1 · 2024-10-15T14:03:37Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/2/execution/node/359/log

daosbuild1 · 2024-10-15T14:08:03Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/2/execution/node/371/log

Signed-off-by: Johann Lombardi <[email protected]>

daosbuild1 · 2024-10-15T14:16:46Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/3/execution/node/387/log

daosbuild1 · 2024-10-15T14:16:48Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/3/execution/node/383/log

daosbuild1 · 2024-10-15T14:17:33Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/3/execution/node/373/log

daosbuild1 · 2024-10-15T14:18:07Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/3/execution/node/374/log

daosbuild1 · 2024-10-15T14:19:11Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/3/execution/node/365/log

daosbuild1 · 2024-10-15T14:23:16Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/3/execution/node/342/log

Signed-off-by: Johann Lombardi <[email protected]>

daosbuild1 · 2024-10-15T14:37:56Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/4/execution/node/387/log

daosbuild1 · 2024-10-15T14:38:08Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/4/execution/node/385/log

daosbuild1 · 2024-10-15T14:38:10Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/4/execution/node/360/log

johannlombardi · 2024-10-17T20:27:06Z

adding @mjmac for the daos cmd output changes.

mjmac · 2024-10-17T20:39:10Z

src/control/cmd/daos/container.go

@@ -962,8 +962,8 @@ func printContainerInfo(out io.Writer, ci *daos.ContainerInfo, verbose bool) err
 			{"Pool UUID": ci.PoolUUID.String()},
 			{"Container redundancy factor": fmt.Sprintf("%d", ci.RedundancyFactor)},
 			{"Number of open handles": fmt.Sprintf("%d", ci.NumHandles)},
-			{"Latest open time": fmt.Sprintf("%#x (%s)", ci.OpenTime, daos.HLC(ci.OpenTime))},
-			{"Latest close/modify time": fmt.Sprintf("%#x (%s)", ci.CloseModifyTime, daos.HLC(ci.CloseModifyTime))},
+			{"Latest open time": fmt.Sprintf("%s (%#x) (%s)", daos.HLC(ci.OpenTime), ci.OpenTime)},


There are three formatting directives, but only two values, unless I'm missing something...

mjmac · 2024-10-17T20:42:04Z

src/control/cmd/daos/snapshot.go

+	cmd.Infof("%s\t\t\t%s\t\t\t%s", "Timestamp" , "Epoch", "Name")
+	cmd.Infof("---------\t\t\t-----\t\t\t----")


For consistency with other commands, it would be better to use the txtfmt package to format these. See printAttributes() for an example.

mjmac · 2024-10-17T20:43:44Z

src/control/cmd/daos/snapshot.go

 	if len(snaps) == 0 {
 		cmd.Info("no snapshots")
 		return nil
 	}
 	for _, snap := range snaps {
-		cmd.Infof("0x%x %s", snap.Epoch, snap.Name)
+		cmd.Infof("%s\t0x%x\t%s", snap.Timestamp, snap.Epoch, snap.Name)


What does the JSON output look like? Is there any processing that needs to happen to make that output useful?

The json output looks good and already has the timestamp, epoch and name.

daosbuild1 · 2024-10-17T20:50:36Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15321/9/testReport/

mchaarawi

looks good to me. but what is the plan for testing this feature?

mchaarawi · 2024-10-17T20:59:49Z

src/include/daos_fs.h

+ * \param[in]	epoch	Epoch associated with the snapshot to mount.
+ *			If a null epoch is passed, then the snapshot is looked up by \a name.


replace null with DAOS_EPOCH_MAX

I am using DAOS_EPOCH_MAX internally already to mean the tip of the container.

no i mean the comment here "If a null epoch is passed". did you mean if 0 epoch is passed?

yes, i meant a epoch of value 0. Null and zero is the same, no? In any case, i can change that.

daltonbohning · 2024-10-17T21:38:49Z

looks good to me. but what is the plan for testing this feature?

Quick suggestion if we want to use ftest.

IOR, e.g. to write some data
mount dfuse on snapshot
IOR read-verify data in dfuse
Write more data to container
IOR read-verify data from dfuse is still the same

Related, if someone uses the datamover tools with a snapshot-dfuse mount as the source, I think the datamover utils will open the pool/cont NOT at the snapshot. Would need to verify

johannlombardi · 2024-10-18T05:44:00Z

the plan for the testing is to extend the dfs test to have some coverage for the new API and then add a new ftest to run tests like suggested by dalton. I just would like to do that in a different PR because i know it is going to take me forever to complete due to the lack of jenkins access.

Signed-off-by: Johann Lombardi <[email protected]>

daosbuild1 · 2024-10-18T06:27:20Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/10/execution/node/374/log

daosbuild1 · 2024-10-18T06:27:32Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/10/execution/node/372/log

daosbuild1 · 2024-10-18T06:28:30Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/10/execution/node/343/log

daosbuild1 · 2024-10-18T06:29:11Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/10/execution/node/344/log

daosbuild1 · 2024-10-18T06:30:17Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/10/execution/node/338/log

daosbuild1 · 2024-10-18T06:34:13Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15321/10/execution/node/387/log

Signed-off-by: Johann Lombardi <[email protected]>

ashleypittman · 2024-10-18T08:09:17Z

Related, if someone uses the datamover tools with a snapshot-dfuse mount as the source, I think the datamover utils will open the pool/cont NOT at the snapshot. Would need to verify

As this is still work-in-progress it would be an idea to just disable the ioctl in this PR, the ioctl will need updating to support this anyway and disabling it would at least prevent any incorrect behaviour by tools built on top of it.
https://github.com/daos-stack/daos/blob/master/src/client/dfuse/ops/ioctl.c#L28

mjmac

Go changes look good to me, thanks.

src/client/dfs/mnt.c

Features: dfs dfuse Signed-off-by: Johann Lombardi <[email protected]>

kjacque · 2024-11-15T01:41:53Z

src/control/cmd/daos/snapshot.go

@@ -211,6 +215,32 @@ func listContainerSnapshots(ap *C.struct_cmd_args_s, containerID string) ([]*sna
 	return snapshots, nil
 }

+func printSnaps(out io.Writer, snaps []*snapshot) {


This function looks good, but would be nice to have a unit test for this output. I would suggest shifting this function into the cmd/daos/pretty subpackage and reference the existing unit tests for verifying output.

kjacque

My other comment is nonblocking. Thanks!

daosbuild1 · 2024-11-15T04:10:49Z

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15321/13/testReport/

johannlombardi added 2 commits October 15, 2024 15:43

DAOS-15971 dfs: add support for mounting a snapshot

41be905

Add new dfs interface to mount a specific snapshot. Provide dfuse command line options to use this feature. Signed-off-by: Johann Lombardi <[email protected]>

Merge branch 'master' into jlo/dfs_snap

a73267e

Signed-off-by: Johann Lombardi <[email protected]>

johannlombardi requested review from a team as code owners October 15, 2024 13:50

fix typos and bump api version.

cbf7758

Signed-off-by: Johann Lombardi <[email protected]>

johannlombardi requested review from a team as code owners October 15, 2024 13:57

fix a few nits

6e41ea0

Signed-off-by: Johann Lombardi <[email protected]>

Fix nits

29931b1

Signed-off-by: Johann Lombardi <[email protected]>

johannlombardi requested a review from a team as a code owner October 17, 2024 20:24

johannlombardi requested a review from mjmac October 17, 2024 20:26

mjmac reviewed Oct 17, 2024

View reviewed changes

mchaarawi reviewed Oct 17, 2024

View reviewed changes

Add support for table in the snapshot list.

6214ee1

Signed-off-by: Johann Lombardi <[email protected]>

Fix a few go nits

3091638

Signed-off-by: Johann Lombardi <[email protected]>

mjmac reviewed Oct 18, 2024

View reviewed changes

kjacque previously approved these changes Oct 22, 2024

View reviewed changes

src/client/dfs/mnt.c Show resolved Hide resolved

src/client/dfs/mnt.c Outdated Show resolved Hide resolved

johannlombardi added 2 commits November 14, 2024 12:05

Merge branch 'master' into jlo/dfs_snap

909c063

Address feedback from Kris

7840170

Features: dfs dfuse Signed-off-by: Johann Lombardi <[email protected]>

johannlombardi dismissed kjacque’s stale review via 7840170 November 14, 2024 11:14

johannlombardi requested review from mchaarawi, mjmac, ashleypittman and kjacque November 14, 2024 11:15

ashleypittman approved these changes Nov 14, 2024

View reviewed changes

kjacque reviewed Nov 15, 2024

View reviewed changes

kjacque approved these changes Nov 15, 2024

View reviewed changes

		cmd.Infof("%s\t\t\t%s\t\t\t%s", "Timestamp" , "Epoch", "Name")
		cmd.Infof("---------\t\t\t-----\t\t\t----")

		* \param[in] epoch Epoch associated with the snapshot to mount.
		* If a null epoch is passed, then the snapshot is looked up by \a name.

DAOS-15971 dfs: add support for mounting a snapshot #15321

Are you sure you want to change the base?

DAOS-15971 dfs: add support for mounting a snapshot #15321

Conversation

johannlombardi commented Oct 15, 2024

github-actions bot commented Oct 15, 2024 • edited Loading

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

daosbuild1 commented Oct 15, 2024

johannlombardi commented Oct 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daosbuild1 commented Oct 17, 2024

mchaarawi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daltonbohning commented Oct 17, 2024

johannlombardi commented Oct 18, 2024

daosbuild1 commented Oct 18, 2024

daosbuild1 commented Oct 18, 2024

daosbuild1 commented Oct 18, 2024

daosbuild1 commented Oct 18, 2024

daosbuild1 commented Oct 18, 2024

daosbuild1 commented Oct 18, 2024

ashleypittman commented Oct 18, 2024

mjmac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kjacque left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 15, 2024

github-actions bot commented Oct 15, 2024 •

edited

Loading