[BUG] Not able to publish anything on new APIM using the publisher pipeline

[prateek-rio]

Release version

v5.0.1

Describe the bug

Hi Team ,

We are trying to create a new dev environment from the existing Prod environment we created using the portal. We are able to extract all the configuration from the prod instance . But when we are publishing it we are not getting any results . The pipeline is running successfully though.

Expected behavior

Publisher pipeline should publish all the configuration

Actual behavior

No changes happening in new APIm

Reproduction Steps

create 2 APIM . Create some APIs and product in first one then run the extractor pipleine and the publisher pipeline.

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[waelkdouh]

Please space the logs here so we can take a look. Make sure you scrub them in case they include any sensitive data.

[waelkdouh]

Also take a look at the YouTube videos that we have on the project homepage in case you are not familiar with the tool yet.

[prateek-rio]

Hi Waelkdouh. Out APIM set up in private networking . Is this going to make any difference . We have used the default pipelines provided in the git repo. We are able to run successfull the extractor pipeline . It is extracting all our configuration . But when we are running the publisher pipeline it is also running successfully but not publishing anything . We have also raised a ticket with Azure MS support . If it is possible can we set up a call tommorow to resolve this issue quickly . Also Can you provide you sample prod.config file as we are not putting much there . It has now just the name of APIM .

[waelkdouh]

We manage this an open source project. So you can't open a microsoft support ticket.

When you say the publisher runs but nothing happens, do you mean the created PR didn't include any artifacts? Is it erroring out? We need to understand what you mean by nothing happens.

[prateek-rio]

No, The artifacts are getting successfully created when we run the extractor pipeline in the new repo and also create a PR to merge with master. When we approve the PR it will run the publisher pipeline . we have provided the name of new APIM in the variable group and in the prod config yaml file. This publisher pipeline also shows all green and is completing successfully. But when we check the new APIM it is now updating the APIs and Products or any other details . You can ping me on teams [email protected] we can have a short quick call .

[waelkdouh]

Sorry can't ping you on teams. We will need you to publish your publisher logs here so we can start providing help. Also please check closed github issues as other folks faced similar issues in the past.

[prateek-rio]

2024-01-09T16:47:58.8985047Z ##[debug]Evaluating condition for step: 'Fetch publisher'
2024-01-09T16:47:58.8985817Z ##[debug]Evaluating: SucceededNode()
2024-01-09T16:47:58.8986082Z ##[debug]Evaluating SucceededNode:
2024-01-09T16:47:58.8986673Z ##[debug]=> True
2024-01-09T16:47:58.8986957Z ##[debug]Result: True
2024-01-09T16:47:58.8987233Z ##[section]Starting: Fetch publisher
2024-01-09T16:47:58.8992673Z ==============================================================================
2024-01-09T16:47:58.8992825Z Task : PowerShell
2024-01-09T16:47:58.8992915Z Description : Run a PowerShell script on Linux, macOS, or Windows
2024-01-09T16:47:58.8993031Z Version : 2.232.1
2024-01-09T16:47:58.8993106Z Author : Microsoft Corporation
2024-01-09T16:47:58.8993210Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
2024-01-09T16:47:58.8993343Z ==============================================================================
2024-01-09T16:47:58.9726829Z ##[debug]Using node path: /home/vsts/agents/3.232.1/externals/node20_1/bin/node
2024-01-09T16:47:59.0474439Z ##[debug]agent.TempDirectory=/home/vsts/work/_temp
2024-01-09T16:47:59.0485079Z ##[debug]loading inputs and endpoints
2024-01-09T16:47:59.0488718Z ##[debug]loading INPUT_TARGETTYPE
2024-01-09T16:47:59.0502665Z ##[debug]loading INPUT_FILEPATH
2024-01-09T16:47:59.0504444Z ##[debug]loading INPUT_SCRIPT
2024-01-09T16:47:59.0505961Z ##[debug]loading INPUT_ERRORACTIONPREFERENCE
2024-01-09T16:47:59.0507288Z ##[debug]loading INPUT_WARNINGPREFERENCE
2024-01-09T16:47:59.0509325Z ##[debug]loading INPUT_INFORMATIONPREFERENCE
2024-01-09T16:47:59.0510779Z ##[debug]loading INPUT_VERBOSEPREFERENCE
2024-01-09T16:47:59.0512933Z ##[debug]loading INPUT_DEBUGPREFERENCE
2024-01-09T16:47:59.0514451Z ##[debug]loading INPUT_PROGRESSPREFERENCE
2024-01-09T16:47:59.0516796Z ##[debug]loading INPUT_FAILONSTDERR
2024-01-09T16:47:59.0520157Z ##[debug]loading INPUT_SHOWWARNINGS
2024-01-09T16:47:59.0522132Z ##[debug]loading INPUT_IGNORELASTEXITCODE
2024-01-09T16:47:59.0524488Z ##[debug]loading INPUT_PWSH
2024-01-09T16:47:59.0526643Z ##[debug]loading INPUT_WORKINGDIRECTORY
2024-01-09T16:47:59.0527988Z ##[debug]loading INPUT_RUNSCRIPTINSEPARATESCOPE
2024-01-09T16:47:59.0529343Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2024-01-09T16:47:59.0530520Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2024-01-09T16:47:59.0532409Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2024-01-09T16:47:59.0539468Z ##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
2024-01-09T16:47:59.0540239Z ##[debug]loading SECRET_AZURE_CLIENT_ID
2024-01-09T16:47:59.0541426Z ##[debug]loading SECRET_AZURE_TENANT_ID
2024-01-09T16:47:59.0542471Z ##[debug]loading SECRET_AZURE_CLIENT_SECRET
2024-01-09T16:47:59.0543636Z ##[debug]loading SECRET_AZURE_BEARER_TOKEN
2024-01-09T16:47:59.0544753Z ##[debug]loading SECRET_AZURE_SUBSCRIPTION_ID
2024-01-09T16:47:59.0546370Z ##[debug]loaded 24
2024-01-09T16:47:59.0550723Z ##[debug]Agent.ProxyUrl=undefined
2024-01-09T16:47:59.0553815Z ##[debug]Agent.CAInfo=undefined
2024-01-09T16:47:59.0554156Z ##[debug]Agent.ClientCert=undefined
2024-01-09T16:47:59.0554495Z ##[debug]Agent.SkipCertValidation=undefined
2024-01-09T16:47:59.0570926Z ##[debug]check path : /home/vsts/work/_tasks/PowerShell_e213ff0f-5d5c-4791-802d-52ea3e7be1f1/2.232.1/task.json
2024-01-09T16:47:59.0573996Z ##[debug]adding resource file: /home/vsts/work/_tasks/PowerShell_e213ff0f-5d5c-4791-802d-52ea3e7be1f1/2.232.1/task.json
2024-01-09T16:47:59.0576571Z ##[debug]system.culture=en-US
2024-01-09T16:47:59.0586085Z ##[debug]errorActionPreference=stop
2024-01-09T16:47:59.0588725Z ##[debug]warningPreference=default
2024-01-09T16:47:59.0590078Z ##[debug]informationPreference=default
2024-01-09T16:47:59.0591128Z ##[debug]verbosePreference=default
2024-01-09T16:47:59.0592118Z ##[debug]debugPreference=default
2024-01-09T16:47:59.0593002Z ##[debug]progressPreference=silentlyContinue
2024-01-09T16:47:59.0594948Z ##[debug]showWarnings=false
2024-01-09T16:47:59.0595892Z ##[debug]failOnStderr=true
2024-01-09T16:47:59.0596962Z ##[debug]ignoreLASTEXITCODE=false
2024-01-09T16:47:59.0598721Z ##[debug]workingDirectory=/home/vsts/work/1/s
2024-01-09T16:47:59.0605183Z ##[debug]check path : /home/vsts/work/1/s
2024-01-09T16:47:59.0605512Z ##[debug]targetType=inline
2024-01-09T16:47:59.0606512Z ##[debug]script=Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$VerbosePreference = "Continue"
$InformationPreference = "Continue"

Write-Information "Downloading publisher..."
$publisherFileName = "Linux" -like "win" ? "publisher.win-x64.exe" : "publisher.linux-x64.exe"
$uri = "https://github.com/Azure/apiops/releases/download/v5.0.1/$publisherFileName"
$destinationFilePath = Join-Path "/home/vsts/work/_temp" "publisher.exe"
Invoke-WebRequest -Uri "$uri" -OutFile "$destinationFilePath"

if ("Linux" -like "linux")
{
Write-Information "Setting file permissions..."
& chmod +x "$destinationFilePath"
if ($LASTEXITCODE -ne 0) { throw "Setting file permissions failed."}
}

Write-Host "##vso[task.setvariable variable=PUBLISHER_FILE_PATH]$destinationFilePath"
Write-Information "Execution complete."
2024-01-09T16:47:59.0607485Z ##[debug]runScriptInSeparateScope=false
2024-01-09T16:47:59.0613445Z Generating script.
2024-01-09T16:47:59.0652841Z ##[debug]Processed: ##vso[task.logdetail id=78923f8b-f145-4e57-a870-c0c03bbeb51b;type=command;name=command;]Formatted command: Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$VerbosePreference = "Continue"
$InformationPreference = "Continue"

Write-Information "Downloading publisher..."
$publisherFileName = "Linux" -like "win" ? "publisher.win-x64.exe" : "publisher.linux-x64.exe"
$uri = "https://github.com/Azure/apiops/releases/download/v5.0.1/$publisherFileName"
$destinationFilePath = Join-Path "/home/vsts/work/_temp" "publisher.exe"
Invoke-WebRequest -Uri "$uri" -OutFile "$destinationFilePath"

if ("Linux" -like "linux")
{
Write-Information "Setting file permissions..."
& chmod +x "$destinationFilePath"
if ($LASTEXITCODE -ne 0) { throw "Setting file permissions failed."}
}

Write-Host "##vso[task.setvariable variable=PUBLISHER_FILE_PATH]$destinationFilePath"
Write-Information "Execution complete."
2024-01-09T16:47:59.0654218Z ##[debug]Agent.Version=3.232.1
2024-01-09T16:47:59.0654593Z ##[debug]agent.tempDirectory=/home/vsts/work/_temp
2024-01-09T16:47:59.0654953Z ##[debug]check path : /home/vsts/work/_temp
2024-01-09T16:47:59.0655232Z ========================== Starting Command Output ===========================
2024-01-09T16:47:59.0655542Z ##[debug]which 'pwsh'
2024-01-09T16:47:59.0655901Z ##[debug]found: '/usr/bin/pwsh'
2024-01-09T16:47:59.0656219Z ##[debug]which '/usr/bin/pwsh'
2024-01-09T16:47:59.0656550Z ##[debug]found: '/usr/bin/pwsh'
2024-01-09T16:47:59.0656870Z ##[debug]/usr/bin/pwsh arg: -NoLogo
2024-01-09T16:47:59.0657195Z ##[debug]/usr/bin/pwsh arg: -NoProfile
2024-01-09T16:47:59.0657545Z ##[debug]/usr/bin/pwsh arg: -NonInteractive
2024-01-09T16:47:59.0657875Z ##[debug]/usr/bin/pwsh arg: -Command
2024-01-09T16:47:59.0658260Z ##[debug]/usr/bin/pwsh arg: . '/home/vsts/work/_temp/c36147c2-4308-40c0-822e-ab3092229d67.ps1'
2024-01-09T16:47:59.0658631Z ##[debug]exec tool: /usr/bin/pwsh
2024-01-09T16:47:59.0658960Z ##[debug]arguments:
2024-01-09T16:47:59.0659259Z ##[debug] -NoLogo
2024-01-09T16:47:59.0659569Z ##[debug] -NoProfile
2024-01-09T16:47:59.0659873Z ##[debug] -NonInteractive
2024-01-09T16:47:59.0660249Z ##[debug] -Command
2024-01-09T16:47:59.0660604Z ##[debug] . '/home/vsts/work/_temp/c36147c2-4308-40c0-822e-ab3092229d67.ps1'
2024-01-09T16:47:59.0661252Z [command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/c36147c2-4308-40c0-822e-ab3092229d67.ps1'
2024-01-09T16:47:59.1051195Z ##[debug]Agent running environment resource - Disk: available:26306.00MB out of 85160.00MB, Memory: used 26MB out of 6921MB, CPU: usage 12.14
2024-01-09T16:48:00.1030755Z Downloading publisher...
2024-01-09T16:48:00.2337810Z �[33;1mVERBOSE: GET with 0-byte payload�[0m
2024-01-09T16:48:01.1234793Z �[33;1mVERBOSE: received 77265990-byte response of content type application/octet-stream�[0m
2024-01-09T16:48:01.5258706Z Setting file permissions...
2024-01-09T16:48:01.5408465Z ##[debug]Processed: ##vso[task.setvariable variable=PUBLISHER_FILE_PATH]/home/vsts/work/_temp/publisher.exe
2024-01-09T16:48:01.5409375Z Execution complete.
2024-01-09T16:48:01.5429447Z ##[debug]$LASTEXITCODE: 0
2024-01-09T16:48:01.6003634Z
2024-01-09T16:48:01.6009369Z ##[debug]Exit code 0 received from tool '/usr/bin/pwsh'
2024-01-09T16:48:01.6014439Z ##[debug]STDIO streams have closed for tool '/usr/bin/pwsh'
2024-01-09T16:48:01.6139989Z ##[section]Finishing: Fetch publisher

[prateek-rio]

Uploading logs_38050.zip…

[waelkdouh]

We need to see what you have here

[prateek-rio]

Starting: Run publisher for Prod environment

Task : PowerShell
Description : Run a PowerShell script on Linux, macOS, or Windows
Version : 2.232.1
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell

Generating script.
========================== Starting Command Output ===========================
/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/93761a42-e78c-4a2f-9d7c-c5cbd08a9abf.ps1'
info: Publisher[0]
Beginning execution...
info: Publisher[0]
Getting files from commit ID 111e9de6fa636a6ee3c1a223e243148929feb624...
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
Content root path: /home/vsts/work/1/s
info: Publisher[0]
Execution complete.
info: Microsoft.Hosting.Lifetime[0]
Application is shutting down...
Execution complete.

Finishing: Run publisher for Prod environment

[jagratimodi]

I am facing the same issue. My QA configuration changes are not getting applied on APIM instance.

[waelkdouh]

Getting files from commit ID 111e9de6fa636a6ee3c1a223e243148929feb624

Can you please confirm what changes live under that commit ID?

[prateek-rio]

logs_38050.zip

[waelkdouh]

The publisher is not finding any changes in the highlighted commit id below. hence its not publishing anything.

You have to make sure that there are changes under the commit id in question. This is why I asked you to post the changes under that specific id. Something that would look like this:

[prateek-rio]

Hi. Is it mandatory to push the changes to both dev and prod environment from the publisher pipeline. Because as of now we have specified the same APIM name in the variables i.e. Prod one that is empty configuration.

[waelkdouh]

No. You are at liberty to modify your pipeline the way you see fit. For example you can avoid publishing to dev and publish to prod directly. We recommend that you deploy to the dev first, even though it feels repetitive since you're extracting from it, just as a safety measure in order to make sure that the extracted artifacts are valid before you publish them to production (or any other environment in between). Of course, the assumption there is that when you are deploying to production, you would be overriding some values (e.g apim instance name, namedvalues, etc.) using the configuration files as things in production (as well as other environments like QA for example) will be different then the dev environment that you are extracting from.