diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 8ef21076..35c0a0e4 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -11,39 +11,10 @@ Repository structure Due to the fact that we support only the production and the development versions, for now only 2 branchs are present: *master*, and *devel*. -* *master* is the stable branch. Production tags are created starting from this branch. -* *devel* is the development branch. Tags created starting from this branch are subject to a certification process. - -The following diagram highlights the interactions between the branches and the merging and tagging strategy: -![LHCbDIRAC branches](https://docs.google.com/drawings/d/14UPBPGW2R8d7JBO9eHWw2tyD3ApEuUBmlDEFicoBs1U/pub?w=1011&h=726) - -For now, actually there are no tags created yet, so we've always used the HEAD of the branches. - -Repositories -============ - -Developers should have 2 remote repositories (which is the typical GitHub workflow): - -* *origin* : cloned from your private fork done on GitHub -* *upstream* : add it via git remote add upstream and pointing to the blessed repository: git@github.com:DIRACGrid/Pilot.git (or https://github.com/DIRACGrid/Pilot.git using https protocol) +* *master* is the stable branch. +* *devel* is the development branch. Issue Tracking ============== Issue tracking for the project is [here in github](https://github.com/DIRACGrid/Pilot/issues). - - -Code quality -============ - -The contributions are subject to reviews. - -Pylint is run regularly on the source code. The .pylintrc file defines the expected coding rules and peculiarities. - - -Testing -====== - -Unit tests are provided within the source code. Integration, regression and system tests are instead in the tests directory. - - diff --git a/Jenkinsfile b/Jenkinsfile deleted file mode 100644 index 0d183a92..00000000 --- a/Jenkinsfile +++ /dev/null @@ -1,141 +0,0 @@ -#!/usr/bin/env groovy - -/* -This is customized pipeline for running on jenkins-dirac.web.cern.ch -*/ - - -properties([parameters([string(name: 'projectVersion', defaultValue: 'v8.1.0a15', description: 'The DIRAC version to install. Use py2 or py3 syntax'), - string(name: 'Pilot_repo', defaultValue: 'DIRACGrid', description: 'The Pilot repo'), - string(name: 'Pilot_branch', defaultValue: 'devel', description: 'The Pilot branch'), - string(name: 'DIRAC_test_repo', defaultValue: 'DIRACGrid', description: 'The DIRAC repo to use for getting the test code'), - string(name: 'DIRAC_test_branch', defaultValue: 'integration', description: 'The DIRAC branch to use for getting the test code'), - string(name: 'JENKINS_CE', defaultValue: 'jenkins.cern.ch', description: 'The CE definition to use (of DIRAC.Jenkins.ch, see CS for others)'), - string(name: 'modules', defaultValue: '', description: 'to override what is installed, e.g. with https://github.com/$DIRAC_test_repo/DIRAC.git:::DIRAC:::$DIRAC_test_branch'), - string(name: 'pip_install_options', defaultValue: '', description: 'options to pip install (e.g. --index-url=https://lhcb-repository.web.cern.ch/repository/pypi/simple)'), - string(name: 'pilot_options', defaultValue: '', description: 'any pilot option'), - string(name: 'CSURL', defaultValue: 'https://lbcertifdirac70.cern.ch:9135/Configuration/Server', description: 'URL for CS'), - string(name: 'DIRACSETUP', defaultValue: 'DIRAC-Certification', description: 'DIRAC setup'), - string(name: 'PILOTJSON', defaultValue: 'pilot_oldSchema.json', description: 'other option: pilot_newSchema.json'), - string(name: 'wnVO', defaultValue: 'dteam', description: 'other option: gridpp') - ])]) - - -node('lhcbci-cernvm4-03') { - // Clean workspace before doing anything - deleteDir() - - withEnv([ - "PILOTCFG=pilot.cfg", - "DIRACSE=CERN-SWTEST", - "JENKINS_QUEUE=jenkins-queue_not_important", - "JENKINS_SITE=DIRAC.Jenkins.ch", - "DIRAC_PILOT_STAMP=1234567890qwertyuiopasdfghjklzxc", - "DIRACUSERDN=/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=alboyer/CN=819281/CN=Alexandre Franck Boyer", - "DIRACUSERROLE=dirac_user"]){ - - stage('GET') { - - echo "Here getting the code" - - dir("$WORKSPACE/TestCode"){ - sh """ - git clone https://github.com/${params.Pilot_repo}/Pilot.git - cd Pilot - git checkout ${params.Pilot_branch} - cd .. - """ - sh """ - git clone https://github.com/${params.DIRAC_test_repo}/DIRAC.git - cd DIRAC - git checkout ${params.DIRAC_test_branch} - cd .. - """ - echo "Got the test code" - } - stash includes: 'TestCode/**', name: 'testcode' - } - stage('SourceAndInstall') { - - echo "Sourcing and installing" - sh """ - source $WORKSPACE/TestCode/Pilot/tests/CI/pilot_ci.sh - fullPilot - """ - echo "**** Pilot INSTALLATION DONE ****" - - stash includes: 'PilotInstallDIR/**', name: 'installation' - - } - stage('Test') { - echo "Starting the tests" - - parallel( - - "Integration" : { - node('jenkins-dirac-worker01') { - - cleanWs() - - unstash "installation" - unstash "testcode" - - try { - dir(env.WORKSPACE+"/PilotInstallDIR"){ - sh ''' - bash -c "DEBUG=True;\ - source bashrc;\ - source diracos/diracosrc;\ - source \$WORKSPACE/TestCode/Pilot/tests/CI/pilot_ci.sh;\ - downloadProxy;\ - python \$WORKSPACE/TestCode/DIRAC/tests/Workflow/Integration/Test_UserJobs.py pilot.cfg -o /DIRAC/Security/UseServerCertificate=no -ddd" - ''' - } - } catch (e) { - // if any exception occurs, mark the build as failed - currentBuild.result = 'FAILURE' - throw e - } finally { - // perform workspace cleanup only if the build have passed - // if the build has failed, the workspace will be kept - cleanWs cleanWhenFailure: false - } - } - }, - - "Regression" : { - node('jenkins-dirac-worker01') { - - cleanWs() - - unstash "installation" - unstash "testcode" - - try { - dir(env.WORKSPACE+"/PilotInstallDIR"){ - sh ''' - bash -c "DEBUG=True;\ - source bashrc;\ - source diracos/diracosrc;\ - source \$WORKSPACE/TestCode/Pilot/tests/CI/pilot_ci.sh;\ - downloadProxy;\ - python \$WORKSPACE/TestCode/DIRAC/tests/Workflow/Regression/Test_RegressionUserJobs.py pilot.cfg -o /DIRAC/Security/UseServerCertificate=no -ddd" - ''' - } - } catch (e) { - // if any exception occurs, mark the build as failed - currentBuild.result = 'FAILURE' - throw e - } finally { - // perform workspace cleanup only if the build have passed - // if the build has failed, the workspace will be kept - cleanWs cleanWhenFailure: false - } - } - }, - - ) - } - } -} - diff --git a/Pilot/pilotCommands.py b/Pilot/pilotCommands.py index a95ce35c..1723ca83 100644 --- a/Pilot/pilotCommands.py +++ b/Pilot/pilotCommands.py @@ -361,6 +361,7 @@ def _localInstallDIRAC(self): for modules in self.pp.modules.split(","): branch = project = "" elements = modules.split(":::") + url = "" if len(elements) == 3: # e.g.: https://github.com/$DIRAC_test_repo/DIRAC.git:::DIRAC:::$DIRAC_test_branch url, project, branch = elements @@ -516,6 +517,8 @@ def _getBasicsCFG(self): if self.pp.userDN: self.cfg.append('-o /AgentJobRequirements/OwnerDN="%s"' % self.pp.userDN) self.cfg.append("-o /LocalSite/ReleaseVersion=%s" % self.releaseVersion) + # add the installation locations + self.cfg.append("-o /LocalSite/CVMFS_locations=%s" % ",".join(self.pp.CVMFS_locations)) if self.pp.wnVO: self.cfg.append('-o "/Resources/Computing/CEDefaults/VirtualOrganization=%s"' % self.pp.wnVO) @@ -846,6 +849,9 @@ def execute(self): localArchitecture = localArchitecture.strip().split("\n")[-1].strip() cfg.append('-S "%s"' % self.pp.setup) cfg.append("-o /LocalSite/Architecture=%s" % localArchitecture) + + # add the local platform as determined by the platform module + cfg.append("-o /LocalSite/Platform=%s" % platform.machine()) configureCmd = "%s %s" % (self.pp.configureScript, " ".join(cfg)) retCode, _configureOutData = self.executeAndGetOutput(configureCmd, self.pp.installEnv) diff --git a/README.md b/README.md new file mode 100644 index 00000000..3c75132e --- /dev/null +++ b/README.md @@ -0,0 +1,8 @@ +# Dirac Pilots + +The Dirac interware is a software framework for distributed computing, +providing a complete solution to one or more user community requiring access to distributed resources. +Dirac builds a layer between the users and the resources offering a common interface to a number of heterogeneous providers, +integrating them in a seamless manner, providing interoperability, at the same time as an optimized, transparent and reliable usage of the resources. + +The Pilot repository is an independent part of the Dirac system dedicated to the development of Dirac Pilots. \ No newline at end of file diff --git a/README.rst b/README.rst deleted file mode 100644 index b896128b..00000000 --- a/README.rst +++ /dev/null @@ -1,57 +0,0 @@ -.. -*- mode: rst -*- - -.. image:: https://travis-ci.org/DIRACGrid/Pilot.svg?branch=master - :target: https://travis-ci.org/DIRACGrid/Pilot - -.. image:: https://readthedocs.org/projects/pilot/badge/?version=master - :target: http://pilot.readthedocs.io/en/master/?badge=master - :alt: Documentation Status - -DIRAC Pilots -============= -.. image:: https://img.shields.io/coveralls/DIRACGrid/Pilot/master.svg?maxAge=2592000 - :target: https://coveralls.io/github/DIRACGrid/Pilot -.. image:: https://landscape.io/github/DIRACGrid/Pilot/master/landscape.svg?style=flat - :target: https://landscape.io/github/DIRACGrid/Pilot/master - :alt: Code Health - -Jenkins Status -~~~~~~~~~~~~~~ -.. image:: https://jenkins-lhcb-core-soft.web.cern.ch/buildStatus/icon?job=DIRAC%20PILOT - :target: https://jenkins-lhcb-core-soft.web.cern.ch/job/DIRAC%20PILOT/ - - - -DIRAC (Distributed Infrastructure with Remote Agent Control) INTERWARE is a software framework for distributed computing -providing a complete solution to one or more user community requiring access to distributed resources. -DIRAC builds a layer between the users and the resources offering a common interface to a number of heterogeneous providers, -integrating them in a seamless manner, providing interoperability, at the same time as an optimized, transparent and reliable usage of the resources. - -DIRAC has been started by the `LHCb collaboration `_ who still is part of the maintainers group. -It is now used by several communities (AKA VO=Virtual Organizations) for their distributed computing workflows. - -The Pilot repository is an independent part of the DIRAC system dedicated to the development of DIRAC Pilots. - - -Important links -=============== - -- Official source code repo: https://github.com/DIRACGrid/Pilot -- Support Mailing list: https://groups.google.com/forum/#!forum/diracgrid-forum -- Developers Mailing list: https://groups.google.com/forum/#!forum/diracgrid-develop - -Development -=========== - - -Code quality -~~~~~~~~~~~~ - -The contributions are subject to reviews. - -Pylint is run regularly on the source code. The .pylintrc file defines the expected coding rules and peculiarities (e.g.: tabs consists of 2 spaces instead of 4) - -Testing -~~~~~~~ - -Unit tests are provided within the source code. Integration, regression and system tests are instead in the tests directory. Run py.test to run all unit tests.