Copy statedb/couchdb recursively discarding non .json files to create index for public and private data assets. #108
Conversation
… index for public and private data assets. Signed-off-by: David F D Reis <[email protected]>
davidfdr
changed the title
Signed-off-by: David F D Reis <[email protected]>
davidfdr
changed the title
There was a problem hiding this comment.
Thanks for the pull request! It would be great to get it merged- there are a few lint errors to fix up, and I've added some tests in PR #109 which aren't passing yet. (I had been wondering about using filepath.Rel(indexSrcDir, src) and the filepath.Match function to get them passing.)
Also, the skip function gets a FileInfo which should simplify things a bit.
Thanks again!
Signed-off-by: David F D Reis <[email protected]>
Signed-off-by: David F D Reis <[email protected]>
internal/util/copy.go
Outdated
| } | ||
|
|
||
| if info.IsDir() { | ||
| return skipFolder(logger, indexSrcDir, src) |
There was a problem hiding this comment.
I think it would be better to check for any errors here instead of just passing them on, e.g.
skip, err := skipFolder(logger, indexSrcDir, src)
if err != nil {
return skip, fmt.Errorf(
"<suitable error message with strings for dir/src being checked>: %w",
...,
err,
)
}
internal/util/copy.go
Outdated
| @@ -83,6 +99,45 @@ func CopyIndexFiles(logger *log.CmdLogger, src, dest string) error { | |||
| return nil | |||
| } | |||
|
|
|||
| // skipFolder checks if the folder will need to be skipped during indexes copy. | |||
| func skipFolder(logger *log.CmdLogger, indexSrcDir, src string) (bool, error) { | |||
| path, _ := filepath.Rel(indexSrcDir, src) | |||
There was a problem hiding this comment.
Needs to handle any errors. I would also be tempted to do this outside the function and just pass in the path, although maybe the error handling would be neater if it stays here. Either would be fine.
internal/util/copy.go
Outdated
| func skipFolder(logger *log.CmdLogger, indexSrcDir, src string) (bool, error) { | ||
| path, _ := filepath.Rel(indexSrcDir, src) | ||
|
|
||
| matchContainsPublicIndexFolder, _ := filepath.Match("index*", path) |
There was a problem hiding this comment.
The pattern should be more specific, i.e. indexes/*, otherwise things like "indexed/" would also match.
internal/util/copy.go
Outdated
| path, _ := filepath.Rel(indexSrcDir, src) | ||
|
|
||
| matchContainsPublicIndexFolder, _ := filepath.Match("index*", path) | ||
| matchContainsPrivateDataCollectionFolder, _ := filepath.Match("collections*", path) |
There was a problem hiding this comment.
Similarly collections/* although that's what you have for matchPrivateDataCollectionFolder. I don't think you need both unless I've misunderstood.
There was a problem hiding this comment.
The first folder after couchdb can be only collections or indexes.
internal/util/copy.go
Outdated
| matchContainsPublicIndexFolder, _ := filepath.Match("index*", path) | ||
| matchContainsPrivateDataCollectionFolder, _ := filepath.Match("collections*", path) | ||
| matchPrivateDataCollectionFolder, _ := filepath.Match("collections/*", path) | ||
| matchPrivateDataCollectionIndexFolder, _ := filepath.Match("collections/*/indexes", path) |
There was a problem hiding this comment.
I wonder if this pattern should be, collections/*/indexes/*
There was a problem hiding this comment.
collections/*/indexes/* will permit this collections/somefolderCollectionName/indexes/anotherfolder correct? And this is not good right?
internal/util/copy.go
Outdated
| matchContainsPrivateDataCollectionFolder, _ := filepath.Match("collections*", path) | ||
| matchPrivateDataCollectionFolder, _ := filepath.Match("collections/*", path) | ||
| matchPrivateDataCollectionIndexFolder, _ := filepath.Match("collections/*/indexes", path) | ||
| relativeFoldersLength := len(strings.Split(path, "/")) |
There was a problem hiding this comment.
This is stretching my knowledge of platform specifics a bit (I'm wondering about the / in the match patterns!) but splitting the path using filepath.Separator would probably be better.
Signed-off-by: David F D Reis <[email protected]>
…he root folder Signed-off-by: David F D Reis <[email protected]>
Signed-off-by: David F D Reis <[email protected]>
Related to issue: #106
Problem when private data indexing is needed.