Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a union operator #249

Open
etrain opened this issue Mar 15, 2016 · 1 comment
Open

Create a union operator #249

etrain opened this issue Mar 15, 2016 · 1 comment

Comments

@etrain
Copy link
Contributor

etrain commented Mar 15, 2016

In addition to the gather operator which concatenates horizontally, it would be nice to have a union operator which concatenates veritcally.

This came up in a discussion with @shivaram today around data augmentation.

@shivaram
Copy link
Contributor

Just to add the idea here is that similar to Pipeline.gather we can express data augmentation as something like

val data = CifarLoader(trainLocation)
val featurePipeline = Pipeline.concat {
  RandomPatch() andThen
  RandomFlip() andThen
  Convolver andThen
  Pooler
  ...
}
// Here the RDD that is passed in to the next stage of the pipeline has the union
val solvePipeline = featurePipeline andThen BlockWeightedSolver()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants