Loading image issue cantaloupe.

[qudratullah1061]

Hi,

I hope you are doing great. I am using cantaloupe image server and generated pyramidal tile tiffs as suggested but still when i load image it takes around 4 secods to load for individual image. Like i request 30 images some images takes around 5-7 seconds to load and server choked some times. I am using amazon Large ec2 instance with 32gb of ram. I have enabled derivative cache increased heap size to 10g when initialized but no luck for now to reduce loading time of images. Can you please tell me where is the bottleneck? I'll be very thank full to you if you can provide some clue to me.

Here is the testing url.

https://iiif.blavatnikarchive.org/iiif/2/baf__baf52574ad5d3093f4d995837db4e712ba08d23b90e4dd9ca34f1d9425346012/full/!1000,1000/0/default.jpg

Website url where you can see multiple images loading and response time in network tab.
https://newiiif.blavatnikarchive.org/search/items

Thank you very much.

[adolski]

Hi @qudratullah1061,

Are you sure these TIFFs are pyramidal and tiled? Can you attach one of them to a comment so I can download and take a look at it?

[qudratullah1061]

Hi @adolski

Thanks for your reply. Here is the image. You can download it from this link. Its converted to pyramidal tiled tiff. Please let me know if anything is wrong with this file. I am converting all formats jpg,tif to tiled pyramidal tiff like below file for cantaloupe as source.

https://drive.google.com/file/d/1TBABX9BgISuGv7xCJmaChFfcEpMkfqXx/view?usp=sharing

[adolski]

Thanks @qudratullah1061. This is indeed a pyramidal tiled TIFF.

A request for /full/!1000,1000/0/default.jpg is taking about 120ms on my local desktop PC. This is using 5.0.2 with FilesystemSource and default configuration (Java2dProcessor assigned to handle TIFF).

A couple of things to check:

1. Are you doing anything from the delegate script that might be slow, like calling a web service?
2. If you are using HttpSource or S3Source, is chunking enabled?

[qudratullah1061]

I am using this script. If user requests for full image i am serving 1000x1000 image maximum. Also if user request for info.json or with only image name i am redirecting to custom html page.

require 'java'
class CustomDelegate
# The maximum scale allowed to 1000,1000 to every one.
attr_accessor :context
def pre_authorize(options = {})
#logger = Java::edu.illinois.library.cantaloupe.delegate.Logger
uri = context['request_uri']
parts = uri.split('/')
if !context['full_size']
# Not an image request its a json or viewer request.
if parts[6].downcase =="info.json"
parts[2] = 'iiifv.blavatnikarchive.org'
parts.delete("2");
parts.delete("3");
parts.delete("info.json");
uri = parts.join('/')
return {
'status_code' => 303, # See Other
'location' => uri
}
end
return true
end
true
end

def authorize(options = {})
	uri      = context['request_uri']
	parts    = uri.split('/')
	# First work out the maximum scaling that 1000px longest side gives.
	size = context['resulting_size']
	fullsize = context['full_size']
	w = size['width']
	h = size['height']
	fw = fullsize['width']
	fh = fullsize['height']
	# If max size is within constraints anyway just OK it...
	if w > 1000 || h > 1000
		uri      = context['request_uri']
		parts    = uri.split('/')
		parts[7] = '!1000,1000'
		uri      = parts.join('/')
		return {
			'status_code' => 303, # See Other
			'location' => uri
		}
	end
	true
end

end

Regarding storage i am using s3 buckets with 6tb size of images around (64k) images. Regarding chunking how can i enable it? Need to enable it from cantaloupe or s3 feature?

[adolski]

The script looks fine... for chunking I would start with the default values from the sample configuration file, and tweak from there:

S3Source.chunking.enabled = true
S3Source.chunking.chunk_size = 512K
S3Source.chunking.cache.enabled = true
S3Source.chunking.cache.max_size = 5M

When I run that request again using S3Source with chunking enabled, I see this in the log:

15:58:14.021 [qtp680712932-32] DEBUG e.i.l.c.s.s.HTTPImageInputStream - Downloaded 17 chunks (17054362 (26.55%) of 64240282 bytes); read 64.92% of chunk data; 6 cache hits; 17 cache misses

This is telling me that Cantaloupe had to download 17054362 bytes from the bucket in order to complete the request. So if you are running 30 such requests in parallel, and you want each to complete within 500 ms, you would need (very roughly) 1GB/sec bandwidth between your EC2 and your S3 bucket. Are you sure you have enough bandwidth to achieve your performance goals?

I would also try to put the EC2 and the S3 bucket in the same AWS region, if possible.

Suspecting that bandwidth is a concern here, I would also consider compressing the TIFF tiles (with LZW, JPEG, etc.) so that the files will transfer faster.

[qudratullah1061]

We are using c5.4xlarge EC2 with 16 vCPU, 32GB Memory, 10 Gbps network bandwidth, and 4,750 Mbps EBS bandwidth

I am tweaking with chunk sizes. Thanks.

EC2 and the S3 bucket in the same AWS region.

[qudratullah1061]

Hi @adolski

I have changed configuration and increased Chunk Size to 512MB and Max Chunk Cache Size to 5GB and performance looks good for now for images which are atleast served once.. Still checking performance with other different tests and will let you know regarding final outcomes.

Thank you very much for your help and feedbacks you provided. These were really helpful. Please let me know if you have further advice regarding image server specs or performance to make it more better.