-
Notifications
You must be signed in to change notification settings - Fork 18.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor data layers to avoid duplication of data transformation code #954
Conversation
@GeenuX Thanks for your initial commit, could you also update the ImageDataLayer to use the same DataTransformer? |
Thanks for working on this @GeenuX, it will be great to have the madness that is the current set of DataLayers cleaned up. Since commenting on @sguada's #569 I've changed my mind a bit...I think each of the transformations actually could be done as layers with little or no extra cost in terms of memory/time. The key would be to use @longjon's The current design looks good and would definitely be an improvement over the current state as well, but is slightly less flexible than having each transformation be a layer. We could modify this to use layers later but might end up having to make proto-breaking changes, which isn't ideal. I'll leave it up to @GeenuX to decide if/how to take this suggestion. |
Start the refactoring of the datalayers to avoid data transformation code duplication. So far, only DataLayer has been done.
3ff84d9
to
f6ffd8e
Compare
I've refactored the ImageDataLayer as well. I agree, it would be nice to have the transformations defined as separate layers. I'm not sure |
Refactor data layers to avoid duplication of data transformation code
@GeenuX I agree this is a worthwhile improvement in itself. Thanks for your further work to improve the data layers and make data processing more sane. |
@sguada I will let you decide how to carry on in #569. While the DataTransformer is a step in the right direction, it would be better still to have transformation layers along the lines of #954 (comment). I will follow-up by refactoring |
Don't panic, but this breaks model definitions in dev as pointed out by @jeffdonahue.
Sorry for merging before correcting these issues. |
Sorry about that! Thanks for the fix! |
Refactor data layers to avoid duplication of data transformation code
Refactor data layers to avoid duplication of data transformation code
As discussed in #941, I started refactoring the data layers to avoid the current code duplication of data
transformations.
(crop, mirror, mean file...)
Could @shelhamer or someone else could have a look at it before I go about refactoring the other
data layers?