Performance Capacity: Anyone using this to scan 250,000 files per month?

[SlickDH]

I am looking at this solution to support several Salesforce applications but have concerns over capacity.

Any thoughts are appreciated.

Slick

[dontirun]

The solution isn't meant for high volumes of concurrent scans. You will likely find that implementing a similar solution using Fargate/ECS is more cost effective and performant for a high volume of scans.

[SlickDH]

Hey dontirun - Thanks for this info, for the high-volume concurrent scan concern is there possible choke points within with the container, the buckets, the gateway, ClamAV itself or all of the above? Do you know if its ever been tested with a high volume of concurrent scans?

[SlickDH]

@dontirun

[dontirun]

I believe the first bottlenecks will likely be the EFS performance (which is tunable) and account lambda concurrency limits. There is potential for S3 downloads to be be a network bottleneck, but going to be the case with any solution downloading a lot from S3

[mistvan]

Hi guys, I am testing this solution with default configuration - max RAM 10240MB and got these performance results:

With GENERAL_PURPOSE EFS:
5,6 MB file scanned in 38 secs
2MB file scanned in 30 secs
0,06MB file scanned in 25 secs

With MAX_IO EFS:
5,6 MB file scanned in 46 secs
2MB file scanned in 35 secs
0,06MB file scanned in 28 secs

So not sure this is a suitable performance for many files in a month.
Can we tune it to better results with current lambda, efs setup @dontirun please?

[dontirun]

Is that accounting for lambda cold starts? ClamAV and freshclam must always initialize on a cold start making the first scan take a while longer.

Regardless, I don't know of any performance optimization opportunities given the current setup? Do you have any suggestions?

[mistvan]

Based on cloudwatch metric - I had 1 cold start and 3 uploaded files - small images around 0,2MB - took about 20 secs to scan. What are your result of scanning some images for example?