Huge file amount problem

[Faq]

Problem description:
time passed and all files were stored in upload folder what grow big in size and huge file amount, the same for cache.

Shrine.storages = {
  cache: Shrine::Storage::FileSystem.new('public', prefix: 'uploads/cache'),
  store: Shrine::Storage::FileSystem.new('public', prefix: 'uploads')
}

Solution idea
1. for future make all new files in folders with date name when file uploaded YYYY-MM-DD, like prefix: 'uploads/2022-12-29'
2. But for existing files: make folders by file creation date and move files

Is there a way to do this with shrine plugins?

[benkoshy]

Problem description: time passed and all files were stored in upload folder what grow big in size and huge file amount, the same for cache.

Shrine.storages = {
  cache: Shrine::Storage::FileSystem.new('public', prefix: 'uploads/cache'),
  store: Shrine::Storage::FileSystem.new('public', prefix: 'uploads')
}

Solution idea

1. for future make all new files in folders with date name when file uploaded YYYY-MM-DD, like prefix: 'uploads/2022-12-29'
2. But for existing files: make folders by file creation date and move files

Is there a way to do this with shrine plugins?

For new files:
https://shrinerb.com/docs/plugins/pretty_location

For old files you'll have to iterate and rename. I forgot how to do it, I'll post a link If and when I find it.

[Faq]

I guess for old files something like https://shrinerb.com/docs/changing-location

[jrochkind]

You probably already know this, but just to be sure, you can of course get file creation/modification date from the file system itself, right? That is what I use for cleanup.

[Faq]

@jrochkind would be super helpful if you could share that script. Would help everyone :)

ended up by using 'created_at' column from model.

def generate_location(io, **options)
    record = options[:record]

    directory_name = (record.created_at || Date.current).strftime('%Y-%m-%d')

    "#{directory_name}/#{super}"
  end

and rake task with script from https://shrinerb.com/docs/changing-location

[jrochkind]

@Faq If it is a normal file system, just google "unix delete files older than 7 days", it is a common unixy task. I'm not in front of my work systems to see what we are actually using, but googled and this looks right:

  find /path/to/files* -ctime +7 -exec rm {} \;

That would delete all files created more than 7 days ago. Please verify before using on a production system and blaming me.

If it's, say, on S3, you'd need something else, but S3 also keeps track of creation time on it's own.

[jrochkind]

@Faq Actually the Shrine docs have a solution in ruby too, using shrine api (for files on ordinary file system), see: https://shrinerb.com/docs/storage/file-system#clearing-cache

[jrochkind]

And shrine docs have some advice for S3 too: https://shrinerb.com/docs/storage/s3#clearing-cache