Is HTTPS+X509 missing with uproot?

[bockjoo]

Hi,
I think uproot is missing the capability to read a file over HTTPS using X509:

See this example:

import uproot

print(uproot.__version__) # 5.0.11

fname = 'https://xrootd-local.unl.edu:1094//store/user/AGC/zmumu/RunIIFall15MiniAODv2/ZToMuMu_NNPDF30_13TeV-powheg_M_50_120/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/20000/022FAAEA-1BB9-E511-A6DF-44A842CFD5D8.root'
tree = uproot.open(fname)
print ( tree.items ) #  <bound method ReadOnlyDirectory.items of <ReadOnlyDirectory '/' at 0x152380fc7970>>

tree.close()

fname = 'https://xrootd-local.unl.edu:1094//store/mc/RunIISummer20UL18NanoAODv9/TTTo2L2Nu_TuneCP5_13TeV-powheg-pythia8/NANOAODSIM/106X_upgrade2018_realistic_v16_L1v1-v1/130000/3094251D-BAB4-6446-86F1-2F9C4D8F339D.root'
try:
    tree = uproot.open(fname)
except Exception as e:
    print (e) # remote server responded with status 403, rather than 206 (range requests) for URL https://xrootd-local.unl.edu:1094//store/mc/RunIISummer20UL18NanoAODv9/TTTo2L2Nu_TuneCP5_13TeV-powheg-pythia8/NANOAODSIM/106X_upgrade2018_realistic_v16_L1v1-v1/130000/3094251D-BAB4-6446-86F1-2F9C4D8F339D.root
    print (type(e)) # <class 'OSError'>

I can not find any import ssl or from ssl in uprtoo is why I think HTTPS+X509 feature is missing.
Typically, to open and read file over the HTTPS with X509, we do something like this:

import json,os,time
import urllib.request, urllib.error
import ssl
import os.path


url = 'https://cms-cric.cern.ch/api/accounts/user/query/?json&preset=people'
url = "https://cmsio9.rc.ufl.edu:1094/store/user/bockjoo/nano_dy.root"

CERTIFICATE_CRT = '/home/bockjoo/.cmsuser.proxy'
CERTIFICATE_KEY = '/home/bockjoo/.cmsuser.proxy'
try:
            myContext = ssl.SSLContext()
            myContext.load_cert_chain(CERTIFICATE_CRT,
                                      CERTIFICATE_KEY)
            with urllib.request.urlopen(url,
                                        context=myContext) as urlHandle:
                urlCharset = urlHandle.headers.get_content_charset()
                if urlCharset is None:
                    urlCharset = "utf-8"
                try:
                 myData = urlHandle.read().decode( urlCharset )
                except:
                 myData = urlHandle.read()
            #response = requests.get(url, headers=headers) #, context=myContext)
except:
            print("Failed to fetch CMS people information from CRIC")
            raise

print ( type (myData) ) 

Thanks,
Bockjoo

[nsmith-]

Hi,

You can do

import ssl

import uproot

proxy_path = "x509up_uxxxx"
url = "https://xrootd-local.unl.edu:1094//store/mc/RunIISummer20UL18NanoAODv9/TTTo2L2Nu_TuneCP5_13TeV-powheg-pythia8/NANOAODSIM/106X_upgrade2018_realistic_v16_L1v1-v1/130000/3094251D-BAB4-6446-86F1-2F9C4D8F339D.root"

ctx = ssl.create_default_context()
ctx.load_cert_chain(proxy_path, proxy_path)

with uproot.open(url, ssl=ctx) as file:
    print(file["Events"].num_entries)

The general idea is that any arguments that uproot is not aware of in uproot.open(..., **kwargs) are passed as storage_options to the eventual FSSpecSource. There, I had some trouble finding documentation on what storage options are available for http source, eventually I understand that it is any keyword arguments to aiohttp.ClientSession.request, and there we find ssl as an optional argument.

[nsmith-]

See #1248 for how to do this in a distributed context