Picasso Rococo halted after chain upgrade

[JafarAz]

You completed the Parachain Tutorials for your parachain and tested locally?

• Yes, we have successfully tested our chain on rococo-local.

Team Name

Composable Finance

Parachain Name

Picasso

ParaID

2087

Parachain Manager Account

5EyK95UCZaWbD3YUvvQKgaSYeP9L6rhd6ddi25xLt3crPCBX

Is this for a short or long term slot?

Long-term: we have secured a slot on Polkadot or Kusama.

Upload your Genesis Head (genesis state) - Do not submit a compressed file.

Upload your Validation Code (genesis runtime Wasm) - Do not submit a compressed file.

(Optional) Upload your Chain Specification

Anything else we should know?

Whilst testing a runtime upgrade on Picasso rococo, the chain panicked on migration and has now halted.

We need the help of the relaychain maintainer to update the parachain code on the relay.

What are the next steps for us to unbrick our rococo chain?

Register your Rococo parachain details for polkadot-js/apps

• Polkadot-js Apps UI PR completed
  • Available here: {Please fix me}

[paritytech-onboard-rococo-p-chain]

Parachain is now being onboarded.

[AlexD10S]

What is the error?

[JafarAz]

The chain panicked on migration

The latest block shows the following:

parachainSystem.ValidationFunctionApplied The validation function was applied as of the contained relay chain block number.

[dzmitry-lahoda]

@kkast can confirm weather we tested migration on chopstick either (rococo + picasso-rococo configs). afaik we usually do so.

[al3mart]

Based on your issue description, we will need you to share with us the new wasm and head you want us to update the relay storage with, once we have that we can quickly move on 👍

[JafarAz]

Is the plan to revert to some block and the same wasm file for the runtime upgrade we planned?

Or do you need a new genesis wasm and new genesis state files

[kkast]

@al3mart @AlexD10S could you please let us know if it is possible to revert to some of our previous state, or we need to basically repeat the process of onboarding by prodiving new wasm and genesis state?

[JafarAz]

@al3mart hey sir, we just need some clarity on the above to send you the files please

[SBalaguer]

@JafarAz we'll force on the relay chain the chain head and wasm blob that work so that you can continue producing blocks, however you want to continue (genesis, block at height X, whatever you'd like). But we need those two to be able to help out :))

[kkast]

@SBalaguer does the chain head have to be the same as precious, or we need to generate a new one? If it is the old one, is it possible to retrieve in your end, I don't think we kept the head we used initially. Or should we generate as when onboarding from scratch?

[dzmitry-lahoda]

@kkast if feels if start from specific block, need to have data from that block, so if you run the first block before fail. would it be the case need to "remove" that block from state in local storage? or it will not be sync - because that final failed block validated as final.

[kkast]

i think we need more to revert more blocks, because the upgrade was scheduled, so we can take 1000 blocks back. but how do i get head for a particular block? @dzmitry-lahoda

[dzmitry-lahoda]

head is just hash and block number. sure you can read it from node storage. i am not sure we have indexer. also you may explorer blocks on rococo. and query store at block. parachain system or para pallets i guess to check. so how you are going to clean storage of node some blocks back to head? because head in store will be in future. would that work at all?

[kkast]

@SBalaguer @AlexD10S @al3mart could you please use this wasm and change the storage to block 1,244,000, we crashed at block 1,246,381. Please let me know if you need anything else.
picasso_testfast_runtime_v10039(1).zip

[JafarAz]

Just bumping on this sirs. Please let us know if the above is sufficient to unbrick Picasso rococo.

[JafarAz]

gm sirs, kind reminder on this CC @al3mart

[al3mart]

Hey! sorry for the delay here.

Could you please also send the state you need to be set in the relay for that specific block as well ? You should be able to retrieve in the very same way than the code.

Just in case, you need to run the following command with the correct params

./target/release/parachain-template-node export-state --chain=<your running chain spec> --base-path <collator db path> 1244000 > chain-exported-1244000.json

This produces a chain spec from which you can take the state and code as if you were to register a parachain for an oboarding. If you can add the state that would be lovely :)

[kkast]

chain-exported-1244000.zip
@al3mart here it is. let me know if it works

[al3mart]

New code and state have been set now

[kkast]

Thank you very much! how much time will it take to update? do we need to do anything with our nodes?

[al3mart]

As long as you are running the same code and building on top of the provided head state, you should see blocks being produced immediately

[kkast]

our nodes are at the older state and ther rtu happened which caused error. how can we repoint to new state?

[al3mart]

Right, first we need to consider if you guys want to maintain block history or not.

In case you are not interested in keeping block history you can just "hard spoon". Run your nodes fresh with the chain spec you have shared here, that would make it. But it would start building blocks with the state of block #1244000 as its genesis.