[v8.1] [diracx] Get token and run integration tests

[chaen]

This should be transparent :-)
It starts integrating diracx by:

• enabling/disabling feature with env variables
• allow it to run in the integration_tests

One thing could be disruptive though:(@fstagni ) The CSApi does not look anymore in Default or <Setup> for the shifter.
What's your plan for that migration ? Tell the people to copy everything under /Default to the root of the Operatons section before upgrading ?

BEGINRELEASENOTES
*CS
CHANGE: do not look for the shifter under the Default or section

*Test
NEW: allow integration tests to run against DiracX

*Core

NEW: get a token for a proxy

*WMS
NEW: call the diracx Job Monitoring endpoint

ENDRELEASENOTES

[fstagni]

On this:

One thing could be disruptive though:(@fstagni ) The CSApi does not look anymore in Default or <Setup> for the shifter. What's your plan for that migration ? Tell the people to copy everything under /Default to the root of the Operatons section before upgrading ?

First of all, why would you need to change that?

We said to remove the "Setup" concept, and several steps have been done in this direction (see notes for https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-8.1). But, the changes up to now have been done only at the database level, and no changes in CS have been planned. "Setup" is everywhere in CS and no detailed plan have been drawn, and really the latest updates were in #5287. For this release we were not planning CS changes as the complexity of upgrading to v8.1 is already significant, but if there's a really good reason...

[fstagni]

This condition is already applied in src/DIRAC/WorkloadManagementSystem/Client/JobMonitoringClient.py.

In general, I think that we can put all this code in src/DIRAC/WorkloadManagementSystem/Client/JobMonitoringClient.py.

[chaen]

This is needed because otherwise sphinx crashes. diracx is not a dependency of DIRAC so we have no other choice that hide all the diracx code behind env variables

[andresailer]

Can't you add diracx to

DIRAC/docs/diracdoctools/__init__.py

Line 8 in 4d534a3

DIRAC_DOC_MOCK_LIST = [

?

How does sphinx crash?

[chaen]

I'll try thanks !

[chaen]

On this:

One thing could be disruptive though:(@fstagni ) The CSApi does not look anymore in Default or <Setup> for the shifter. What's your plan for that migration ? Tell the people to copy everything under /Default to the root of the Operatons section before upgrading ?

First of all, why would you need to change that?

When running the export script, it crashes without that, as there are no setup known

We said to remove the "Setup" concept, and several steps have been done in this direction (see notes for https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-8.1). But, the changes up to now have been done only at the database level, and no changes in CS have been planned. "Setup" is everywhere in CS and no detailed plan have been drawn, and really the latest updates were in #5287. For this release we were not planning CS changes as the complexity of upgrading to v8.1 is already significant, but if there's a really good reason...

I think that saying that before upgrading to v8.1 it is necessary to copy all the /Operations/<setup/default> to /Operations/<vo> or /Operations is not too much of a change.

[fstagni]

I think that saying that before upgrading to v8.1 it is necessary to copy all the /Operations/<setup/default> to /Operations/<vo> or /Operations is not too much of a change.

That's a bit more than than. Something like

Operations
{
	Defaults
	{
		someOption = A
	}
	someSetup
	{
		someOption = B
	}
	vo
	{
		Defaults
		{
			someOption = C
		}
		someSetup
		{
			someOption = D
		}
	}
}

Should become

Operations
{
	someOption = B
	vo
	{
		someOption = D
	}
}

or even

Operations
{
	vo
	{
		someOption = D
	}
}

all this supposing that someSetup actually exists. So, depending on how much crap is these sections, it might become a lot of manual work. If you make a script though that handle all these cases...

[chrisburr]

We think we still need Default for multi-vo so it can be layered on top of each VO's config without duplicating everything needlessly.

[chaen]

We think we still need Default for multi-vo so it can be layered on top of each VO's config without duplicating everything needlessly.

I am not sure I understand.

What you suggests is the following ?

Operations
{
        Default
        {
	      someOption = B
        }
	vo
	{
		someOption = D
	}
}

CS changes for multi-VO

[fstagni]

We think we still need Default for multi-vo so it can be layered on top of each VO's config without duplicating everything needlessly.

I am not sure I understand.

What you suggests is the following ?

Operations
{
        Default
        {
	      someOption = B
        }
	vo
	{
		someOption = D
	}
}

The above is just to say that what is below is a case:

Operations
{
        Default
        {
	      someOption = B
              other_Option = BBB
              yet_another_Option = BBBBBBB
        }
	vo
	{
		someOption = D
	}
	another_vo
	{
		other_Option = AAA
	}
}

[chaen]

I am now very confused. What I propose is that what is contained in the Setup or Default is promoted one level higher.

Operations
{
	Defaults
	{
		someOptionInGlobolDefault = A
	}
	someSetup
	{
		someOptionInGlobalSetup = B
	}
	vo
	{
		Defaults
		{
			someOptionInVoDefault = C
		}
		someSetup
		{
			someOptionInVoSetup = D
		}
	}
}

becomes

Operations
{
        # Remove default and Setup
	someOptionInGlobolDefault = A
	someOptionInGlobalSetup = B
	vo
	{
               # Remove default and Setup
		someOptionInVoDefault = C
		someOptionInVoSetup = D
	}
}

What is the use case not satisfied with this ?

[fstagni]

If the absence of "Defaults" is interpreted like today's "Defaults" then I think it's also fine. Operations helper need to of course be updated in v8.0 for that.

[chaen]

I think it even eases the migration because everything can just be copied one level up, prior to migrating to v8.1, so we don't even need to touch the code of v8.0
And then we remove the older section