[BUG] sim/smp is broken

[raiden00pl]

Description / Steps to reproduce the issue

on tag: nuttx-12.7.0-RC0:

./tools/configure.sh sim/smp
 make -j
 
 [raiden00:~/git/RTOS/nuttx/nuttx]$ ./nuttx                                                                                                                                                                                                                        
[1]    332900 segmentation fault (core dumped)  ./nuttx

On the current master (a613bcb) we don't get segmentation fault and we can boot to NSH but it's not functional: interaction with NSH lags like hell.

On which OS does this issue occur?

[OS: Linux]

What is the version of your OS?

Arch Linux

NuttX Version

nuttx-12.7.0-RC0, master

Issue Architecture

[Arch: simulator]

Issue Area

[Area: Kernel]

Verification

• I have verified before submitting the report.

[raiden00pl]

It looks like sim/nsh behaves the same. On the other hand sim/foc works fine.

EDIT: sim/nsh is OK. I must have checked something wrong before. The problem is in sim/smp

[xiaoxiang781216]

@hujun260 please take a look.

[hujun260]

#13933
fix this issue

[raiden00pl]

I'm reopening this because sim/smp is still broken. Now console output doesn't work properly.
Here are the details: #13933 (comment)

[raiden00pl]

another problem is assertion in ostest which happens randomly:

signest_test: With task locking
  Total signalled 2480  Odd=1240 Even=1240
  Total handled   2480  Odd=1240 Even=1240
  Total nested    0    Odd=0   Even=0  
[CPU1] dump_assert_info: Current Version: NuttX  10.4.0 6e12b3d74b Oct 12 2024 10:57:16 sim
[CPU1] dump_assert_info: Assertion failed : at file: signest.c:228 task(CPU1): ostest process: ostest 0x441c23
ostest_main: Exiting with status 256
stdio_test: Standard I/O Check: fprintf to stderr

[raiden00pl]

another issue in ostest is broken wdog_test that produce garbage output

wdog_test start...
[CPU2] wdtest_once 0 ns
[CPU2] wdtest_o[nCcPeU [31C] P nUsw0
d]te swtd_o[tnCecPesU 2t0] _ nwosdn
tceset _0on[ cCneP sU0
3 ]n sw
dtes[t_CoPncUe0 0]  nws[
dCPtUe2s] tw_dotnescte_o n[0cC ePn U0s3 
]n sw
dtest_[onCcPeU 00 ]n s[w
CdPtU2e]s twd_toensc[teC_ Po0Un 3cn]es  w
0d tness
t_[oCnPcUe0 ]0  nwsd
t[eCsPUt2_]o nwcde[tC Pe0Us 3tn]_ sow
dntcees t0_ o[nnCscP
eU 00 ]n sw
[dCtPeUs2t]_ owndctee s0t _nosn
ce 0 ns
[CPU1] wdtest_once 10 ns
[CPU1] wdtest_once 1 ns
[CP[[UCCP2U1P]] Uwd t0ews]td _owtncdeet s1e tsnst
__oon[nCPccU1e]e w  dte11st _ onnncess 1 

ns
[[CCPU1P] wUd[te2sCt]P_oUn ce0 w1] nds 
twde[tCsPeU1ts] wt_dt_eosotn_noncccee e1  n s1
1  n[CsnPU
1s] w
dtest_once 1 ns[
CPU0[]C[ CwPPUd1U]t ew2dst]tes t__owoncdne tc1 nees
s 1t [_CPnUo1s]n 
wdcteest_ on1ce  1 [nnsC
sP
U0] wdtest_once 1 ns
[CPU1] wdtest_on[cCPeU 21] 0w0 dnts
[esCtP[_CUPoU01n]]c  wedw ted1s0tt_ eonnsscet
 1_0 nosn
ce [1C[P0CPU U12n]] sw dtw
edstt_oencse t10_ nos
[ncCeP[ UC1P0U01]]  w ndtwsesd
t_tonecse t10_ nso[
nCcP[UeCP2 U]11]  0wwdd ttenestss_ot
n_ceo 1n0 cnse
 1[0C[ CPnPUsU1
] w0dte]st_ own[cedC 1P0t Un2es
]s twd_[CtPoeU1sn]t wc_dtoeesntc_ oen1 ce1 0100   nsn
ns
s
[CPU1] wdtest[_oCnPceU 210]  nsw
dtest_once 10 ns
[CPU1] wdtest_once 1000 ns
[CPU[2[]CC PPUw1U]d w0tdtee]sstt_ o_ncowe nd1c0t0e n es1
s00t [nC_PsU1o
] wndtecst_eon c[e1 C10P00U n20s]
  wnd[CtPsUe1
s] twd_teosnt_conece  110[00 Cn0s
P nUs0[
CP]U1 ] wwdtdestt_[oenCcesP 1Ut002 _]ns 
ownd[ctCePeU 1s] 1wtdt0_es0to_ nonncce se10
 0 n1s
00 ns[[C
CPU1P] Uwdt0est]_o n[cweC 1dP00t Unse
2s]t [C_wPUo1d] ntwdceteesst _ton1_ce0o 01n0 0 cnnses
 
1[C0PU01]  wndtesst
_once 100 ns
[CPU1] wdtest_once 10000 ns
[CP[UCP2U1]] w dtewstd_[otnCceeP s1U0t000 _]nso
 nwdc[tCePeU1 ]s t1wd_t0eost0n_ocn0cee  1 0n010 0sns0

0 [nCPsU1
] wdtest_onc[e 1C00P0 nUs
2[C]P[ CPUUw01]d] w dttweedstst_oten_cseo t10n_00co nsen
c e1[ CP0U110]0 w00dt 0es tn_nosnsce

 1000 ns
[CP[U[CPCU01P]] wU dt2ewst]d_otn ceew s10dt00_t nose
nscte[C _PU11o]0 nwd0ctes0et_ o nnc1es 01
0000 0ns 
ns[
C[PCPUU10] w]dte stw_ondce t100e0 nss
t_once 1000 ns
[CPU1] wdtest_once 100000 ns
[[CPCU1]P wdUt[e2stC_]onPc e U1w000d0]0t  nsew
sdtt[C_ePUo1s]n twcd_teoes nt_1con0ece0  100100000  0nns0
s0
 n[CsPU
1] wdtes[t_ConPceU 1020[0]0C n Ps
wU0d]t[C PewU1sd] wttdtee_ssto_ton_nccoe ne10c0 e00 1 n1s0
00000[C00PU 1 ]n nswd
ste
st_once 10000 [nCs
PU[0C[]CPP U1Uw] 2wdd]tte set_wsontdce_ to10en00c0s enst 
1_0o[0nCP0Uc10]  ewdnt ses
1t_o0nc0e 01000[0 0C nPnsUs
0
][ CPwU1]d wdtteset_osncte _10o00n0 cnse
 10000 ns
[CPU1] wdtest_once [1C00P0U0200]  nws
d[tCePsUt_0[oC]Pn Ucw1e]d  t1we0dst0te0_s0to0_n ocnnecs e
 110000000000 [n Csn
PsU
2] w[CdPtUe1s][t C_wPodUtne0cs]et  _wo1nd0ct0ee0 s01t000_ 0on0ns0c
 ens 
100[0C0P0U [2nC]PsU 
1w]d twedstte_[sotCn_PocUnec0 e] 1 100w00d00t000e0 s ntsn
_so
nce 1[C0P0U[01C]0P 0wU d2nt]es s
tw_dotnecse t1_[0oC0n0P0cU0e0  ]n1s 
0w0d0t0e0s[ tCn_PsUo1
n]c wed t1e0st[0_C0oPn0Uc0e2  ]n1 s0w0
0d0t0e snts_
o[nCcPeU 01]0 0w0d0t0e snts_
once 100000 ns
[C[P[UCCP0UP1]]U  ww2ddt]ets et_wsontdc_teoe n1sc00et0 0_010o 0nn0s
c00e0 0[1 CPn0U1s0]
0 w0dt0es0t_ onnc[esC 1P
0U0000]00  wnsd
te[sCtP[C_UPoU21n]]c we d twe1sdt0_t0on0ec0es 01t00_0 0o0n0ns0 
cnse
 100[0C0[PC0PUU010 ]] nw dswted
stt_eonscte _1o00n0c0[0e0C  nP1s
0U020]0[ 0CwP0Ud 1]tn wsed
tsestt__onocen 1[c0C0e0P0 0U010 n0]s
 0w0d0t[e0Cs0PtU 1_]no wsndtc
eest _o1n0ce0 01000[000C000P  nnUss
2
] [wCPdUt1]e wsdttes_t_oonncec 1e00 001000 n0s
0000 ns
[CPU1] wd_start with maximum delay, cancel OK, rest 2147483645
[CPU1] wdtest_recursive 1000000us
[CPU1] wd_start with maxi[[mCuCmPP dUUel02a]y],   wcwdan_dcse_tlsa OrtKta,  rwrteis tt wh2i 1mt4a7h4x8 i3m6m4ua5m
x[ iCdPmeU1lu]a myw d,td ecesatln_craeeyclu, rsO iKvc,ea  1nr0ec0s0e0tl0 0 2us1O
4K7,4 8r3e[6sC4Pt5U 1
]2[ wC1dP_4sUt70a4]rt8  3wwid6tth4 em5saxt
i_m[urCme PcdeuUlr2as]y,i  vcwaedn cte1le 0Os0Kt,0 _0r0re0setuc su2
1r4s74i8v36e45 
1[[C0CPU0P1U0] 00w]d0t 0ewusdts__r
sectuarsritv ew 1i0t0h0 0[0mC0auxPsi
Um2u]m  w[ddCeP_lUas1y]t ,wa drc_statn acrwtei lwt ihOthK  ,mm aaxrixemsiutmm  ud2em1la4 y7,d4 e8ca3ln6cae4yl5 ,
O [K,cC ParUens0ct] e 2w1l4d 7t4Oe8sK3t6,4_ 5r
re[eCcPsuU1tr]s  iw2dv1tee4 s7t1_40re08c0u30rs06i0v4eu5 s1

00[0C00P0uUs2
] [CwPdU[tC0eP]Us 1t]w _dwr_d_esstctarautr rwtsi tiwhv iemta hx1i m0mua0mx 0ide0mlu0amy0,  udcaesnlc
aeyl ,O Kc,a rnecste l2[ 14OC7K4P8,3U 64r25e]
s[ CtwP Ud21_]1 s4wd7tt4eas8rt3_t6r e4cw5ui
rst[iChvPe  Um100a]00 x0w0id0mutseu
smt _dree[clCuParUy1s],i wv de_cs at1na0rc0t e0w0li0t h0O umKas,x
i murm edselta y,2 [c1CanP4cUe7l04 ]O8 K3,w 6dr_4ess5tt 
a21r[4tC7 4Pw8iU36t24h5]
  [mCwPaUdx1it]m ewusdmtt ed_setr_leraecycu,ur rscsiavinev c1ee0 0l01 00O000Ku0,s0
 0r0eusst
 [2CP1U417] 4w8d3_s6[t4Ca5rPt
U [w2iC]tPh U wm0da]x_i smwutdmat rdeetsl tawy_,ir techac nucmrealsx iOivKme,u  mr1e s0dt0e 02l104a0740y8u,3s6 
4c5a
n[CcPeUl1 ] OwKd[,tCe PsrtUe_0sr]etc  uw2rds1_i4sv7et4 a81r030t60 04w050i
uts[
hC PmUa2x]i mwudmt edsetl_arye,c ucrasnicveel  1O0K0,0 0r0e0suts 
2147483645
[CPU0] wdtest_recursive 1000000us
[CPU1] recursive wdog triggered 6 times, elapsed tick 12
[CPU1] wdtest_recursive 10000000us
[CPU1] recursive wdog t[r[CiCPgPUgU2e0]r] e rdre ec6cu urtrsisimivevese , w wdedologag p tstreridig gggteeirrceekdd   1662  
tt[iiCmmPeeUss1,,]   eewlldaatppessseetdd_  rtteiicccukkr  s11i22v

e[[ CC1PP0UU0200]]0  0ww0dd0ttueesss
tt__rreeccuurr[ssCiiPvvUee1  ]11 00r00e00c00u00r00s00iuuvsse

 wdog trigg[e[CrCPePUdU 206]]   trrieemcceuusrr,ss iievvleea  pwwsddeoodgg   tttirrciikgg gg1ee2rr
ee[ddC  P66U  1tt]ii mmweedsst,,e  seetll_aarppessceeuddr  sttiiivccekk   1110220

0[[0CC0PP0UU020]u] s w
wddtteesstt_[_rCrePecUcu1ur]rs sirivevece u 1r10s00i000v00e00 00w00duuossg

 triggered [6[C CPtPUiU2m0]e] s r,re ececulurarspsisivevede   wtwdidocogkg   t1tr2ri
ig[ggCgePerUre1ed]d   6w6 d tttieimsmetess_,,r  eeeclluaarppsssieevdde   tt1ii0cc0kk0  0110220

0[[uCCsPP
UU20]]  wwdd[ttCeePssUtt__1rre]ec cururerscsiuivrves ie1v 0e10 0w00d00o00g00 0ut0sr0
iugsg
ered 6[ CtPiUm0e[]sC ,Pr eeUcl2u]arp ssrieevdce u trwidscokig  v1te2r 
iw[gdCgoPegUr 1e]td  rw6id tgtegisemtre_sre,ed c eu6lr astpiisvmeeed s 1t,0i 0c0ek0 l01a020p
us[seC
PdU 0t]i wcd[ktC eP1sU2t1
_][r CerPceUuc2ru]sr isviwevd et1 e0w0ds0ot0g_0 r0te0rcuiusgr
gseirveed  61 [0tC0iP0mUe00s]0, 0 r0eelucausprs
seidv et iwcdko [g1 2Ct
Pr[UiC2gPgU]e1 r]re edwc du6tr estsiitvm_eerse ,cw uderolsaigpv sete rd1i 0gt0gi0ec0rk0e 0d10 2u6
s [
tCiPmUe0]s ,w detleaspts_eredc utriscikv e1 21
0[0C0P0U020]0 uwsd
test_recursive 10000000us
[CPU1] recursive wdog triggered 6 times, elapsed tick 12
[CPU2] recursive wd[[oCCgPP UUt03r]]i  [grrCgeePeccUuru1rer]ssd ii rvve6eec   uwtwrddisooimggv e etts rrw,iidg goggegee lrrteaerddpi  gs66g e etdtri iemtemdsie ,sc6 ,k e t leial1mpae2sps
es,de  edtl iatcpiks [ce1Ckd2P  
Ut12i]2c k
r e1c2u
[rCsPiUv0e]  wrdeo[c[gCuC PrtPUsr1Uii]v3g eg]r e ewrcrdeuoedrgc s 6uit vrrtesii gimwgevdesoer,ge   dewt lrd6aio pgtgsgiee mdrte esrtd,ii  c6gek lg tae1ip2rms
eeesdd,   te6il ca[ktpC isP1me2Ued
2 s]t ,ir ceekcl ua1r2ps
[isCvePedU  0w]td oirg[ce CcktPu rUri11sg]2ig 
verere ecwdud ro6sg i tvtierm iewgsdgo,eg[r  CeetPdlrU ia36g]p g sterieremedcedu s r,t6s  iietvcliekam  pews1sde,2od
 g e tltiarcpiksg eg1d[e2 rC
tePidcU k62  1]t[2i C
mrPeeUsc0,u] r es[rliCeavPcpUeus1r e]sdw i rdvteoeic gcuwk rd sto1irg2v i
etg rwgidgeogrge ertderd i g66g  ettriiemdme es6s,  ,tei lmeaelpsas,pe dse letaidpsce kdt  i1tc2ik
c k 1122

wdog_test end...

[xiaoxiang781216]

look like sim_uart driver forget to protect the concurrent access:
https://github.com/apache/nuttx/blob/master/arch/sim/src/sim/sim_uart.c
we need add protection like the real uart driver.

[hujun260]

another problem is assertion in ostest which happens randomly:

signest_test: With task locking
  Total signalled 2480  Odd=1240 Even=1240
  Total handled   2480  Odd=1240 Even=1240
  Total nested    0    Odd=0   Even=0  
[CPU1] dump_assert_info: Current Version: NuttX  10.4.0 6e12b3d74b Oct 12 2024 10:57:16 sim
[CPU1] dump_assert_info: Assertion failed : at file: signest.c:228 task(CPU1): ostest process: ostest 0x441c23
ostest_main: Exiting with status 256
stdio_test: Standard I/O Check: fprintf to stderr

if revert one commit (sim: fix context-switch when do wdog callback()) signest_test looks good
git revert a3568af

[jerpelea]

backported #13968

[jerpelea]

@raiden00pl please check releases/12.7

[jerpelea]

git revert a3568af

type something and it will still fail
./nuttx

NuttShell (NSH) NuttX-12.7.0
nsh>
nsh> Segmentation fault

[raiden00pl]

@jerpelea I get the same result as yours with "Segmentation fault" for release/12.7. But sim/nsh works fine.

The branch master still has not solved the problem with missing characters on output

[raiden00pl]

here backtrace from GDB for sim/smp crash on release/12.7:

pwndbg> bt
#0  0x00007ffff7d371d8 in pthread_sigmask () from /usr/lib/libc.so.6
#1  0x0000000000459a8d in up_irq_restore (flags=0) at sim/posix/sim_hostirq.c:118
#2  0x00000000004081d0 in leave_critical_section (flags=0) at irq/irq_csection.c:422
#3  0x0000000000405b2b in sched_unlock () at sched/sched_unlock.c:206
#4  0x000000000040445e in sim_unlock () at sim/sim_doirq.c:44
#5  0x000000000045a355 in sim_idle_trampoline (arg=0x7fffffffda00) at sim/posix/sim_hostsmp.c:109
#6  0x00007ffff7d2fded in ?? () from /usr/lib/libc.so.6
#7  0x00007ffff7db30dc in ?? () from /usr/lib/libc.so.6

[xiaoxiang781216]

@hujun260 please take a look.

[hujun260]

@hujun260 please take a look.
try add this pr to release/12.7
#13753

[raiden00pl]

@jerpelea damn, I accidentally did a git push to release/12.7 branch with this change #13753
Shouldn't git push to upstream repo be locked somehow in github settings?

[raiden00pl]

with this change #13753 sim/smp boots.
But I just get crash with ostest:

user_main: pthread_rwlock_cancel test
pthread_rwlock_cancel: Starting test
[CPU1] dump_assert_info: Current Version: NuttX  12.7.0 1a3537bcab Oct 16 2024 11:26:52 sim
[CPU1] dump_assert_info: Assertion failed tcb->cpu == up_cpu_index(): at file: sched/sched_removereadytorun.c:293 task(CPU1): ostest process: ostest 0x441715
[CPU1] dump_assert_info: Current Version: NuttX  12.7.0 1a3537bcab Oct 16 2024 11:26:52 sim
[CPU1] dump_assert_info: Assertion failed tcb->cpu == up_cpu_index(): at file: sched/sched_removereadytorun.c:293 task(CPU1): ostest process: ostest 0x441715
[1]    2493755 killed     ./nuttx

[hujun260]

missing some pr
#13731
#13734
#13728 one conflict resolution can refer to the code in the master branch. make void nxtask_exithook(FAR struct tcb_s *tcb, int status) same as master branch

[jerpelea]

@jerpelea damn, I accidentally did a git push to release/12.7 branch with this change #13753 Shouldn't git push to upstream repo be locked somehow in github settings?

I can not see your commit :). I thin that you pushed to your copy

[jerpelea]

@raiden00pl
backported
#14380

local test
NuttShell (NSH) NuttX-12.7.0
nsh> ls
/:
dev/
proc/
nsh>

[jerpelea]

@raiden00pl please test and close the bug

[raiden00pl]

The problem with strange console output is still not solved (probably random missing newline character):

NuttShell (NSH) NuttX-12.7.0
nsh> nsh> help

or:

NuttShell (NSH) NuttX-12.7.0
nsh> help
help usage:  help [-v] [<cmd>]


    .           cp          exit        ls          ps          true        
    [           cmp         expr        mkdir       pwd         truncate    
    ?           dirname     false       mkrd        rm          uname       
    alias       dd          fdinfo      mount       rmdir       umount      
    unalias     df          free        mv          set         unset       
    basename    dmesg       help        pidof       sleep       uptime      
    break       echo        hexdump     poweroff    source      usleep      
    cat         env         kill        quit        test        xd          
    cd          exec        pkill       printf      time        wait        

Builtin Apps:
    getprime    nsh         sh          taskset     
    hello       ostest      smp         
nsh> help
help usage:  help [-v] [<cmd>]
  help [-v] [<cmd>]

    .           cp          exit        ls          ps          true        
    [           cmp         expr        mkdir       pwd         truncate    
    ?           dirname     false       mkrd        rm          uname       
    alias       dd          fdinfo      mount       rmdir       umount      
    unalias     df          free        mv          set         unset       
    basename    dmesg       help        pidof       sleep       uptime      
    break       echo        hexdump     poweroff    source      usleep      
    cat         env         kill        quit        test        xd          
    cat         env         kill        quit       cd          exec        pkill       printf      time        wait        

Builtin Apps:

Builtin Apps:
    getprime    nsh         sh          taskset     
    hello       ostest      smp         
nsh> 

it only happens for sim/smp

[hujun260]

git revert a3568af

The problem with strange console output is still not solved (probably random missing newline character):

NuttShell (NSH) NuttX-12.7.0
nsh> nsh> help

or:

NuttShell (NSH) NuttX-12.7.0
nsh> help
help usage:  help [-v] [<cmd>]


    .           cp          exit        ls          ps          true        
    [           cmp         expr        mkdir       pwd         truncate    
    ?           dirname     false       mkrd        rm          uname       
    alias       dd          fdinfo      mount       rmdir       umount      
    unalias     df          free        mv          set         unset       
    basename    dmesg       help        pidof       sleep       uptime      
    break       echo        hexdump     poweroff    source      usleep      
    cat         env         kill        quit        test        xd          
    cd          exec        pkill       printf      time        wait        

Builtin Apps:
    getprime    nsh         sh          taskset     
    hello       ostest      smp         
nsh> help
help usage:  help [-v] [<cmd>]
  help [-v] [<cmd>]

    .           cp          exit        ls          ps          true        
    [           cmp         expr        mkdir       pwd         truncate    
    ?           dirname     false       mkrd        rm          uname       
    alias       dd          fdinfo      mount       rmdir       umount      
    unalias     df          free        mv          set         unset       
    basename    dmesg       help        pidof       sleep       uptime      
    break       echo        hexdump     poweroff    source      usleep      
    cat         env         kill        quit        test        xd          
    cat         env         kill        quit       cd          exec        pkill       printf      time        wait        

Builtin Apps:

Builtin Apps:
    getprime    nsh         sh          taskset     
    hello       ostest      smp         
nsh> 

it only happens for sim/smp

git revert a3568af can solve this

[jerpelea]

git revert a3568af

The problem with strange console output is still not solved (probably random missing newline character):

NuttShell (NSH) NuttX-12.7.0
nsh> nsh> help

or:

NuttShell (NSH) NuttX-12.7.0
nsh> help
help usage:  help [-v] [<cmd>]


    .           cp          exit        ls          ps          true        
    [           cmp         expr        mkdir       pwd         truncate    
    ?           dirname     false       mkrd        rm          uname       
    alias       dd          fdinfo      mount       rmdir       umount      
    unalias     df          free        mv          set         unset       
    basename    dmesg       help        pidof       sleep       uptime      
    break       echo        hexdump     poweroff    source      usleep      
    cat         env         kill        quit        test        xd          
    cd          exec        pkill       printf      time        wait        

Builtin Apps:
    getprime    nsh         sh          taskset     
    hello       ostest      smp         
nsh> help
help usage:  help [-v] [<cmd>]
  help [-v] [<cmd>]

    .           cp          exit        ls          ps          true        
    [           cmp         expr        mkdir       pwd         truncate    
    ?           dirname     false       mkrd        rm          uname       
    alias       dd          fdinfo      mount       rmdir       umount      
    unalias     df          free        mv          set         unset       
    basename    dmesg       help        pidof       sleep       uptime      
    break       echo        hexdump     poweroff    source      usleep      
    cat         env         kill        quit        test        xd          
    cat         env         kill        quit       cd          exec        pkill       printf      time        wait        

Builtin Apps:

Builtin Apps:
    getprime    nsh         sh          taskset     
    hello       ostest      smp         
nsh> 

it only happens for sim/smp

git revert a3568af can solve this

Hi
a3568af10551c37b839169634c33fbe78044725f is included in the #14380

[raiden00pl]

my mistake. I didn't notice that #14380 PR is not merged. With this PR sim/smp works fine

[jerpelea]

after merge please close the bug

[raiden00pl]

ostest is throwing garbage at wdtest, but I think we can close this issue for now:

wdog_test start...
[CPU1] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU2] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU1] wdtest_once 0 ns
[CPU2] wdtest_[onCce P1 [UnsC0
P]U 1w[]CdP Ut2w]e dwsdtttees_ts_ootnnc_ec o1e nn sc
1e  n1[s CP
Un2]s wd
test_once 1 n[s
CPU0] [wC[dPCPtUU2e]1 sw]dtt e_swto_donntccee e1 sn st
1_ onn[sCcP
Ue2]  wdt1es t_noncse 
1[ nCs
PU0] [wCd[CPPtUU2e]1 sw]dtt e_swto_donntccee 1es n ts1
_ onnsc[Ce
PU 2]1 w dtensts_o
nce [1 nCs
PU0] [wC[CdPPUt2U]e 1wsd]tte s_tw_oodnnctec e1e sn ts
1_ onnsc
e 1 ns
[[CCPUP2]U wd0t[e]sCt _PownUcd1et ]10e  snwst
d_toe[nsCcPtUe2_ ]o 1wnd0ctee s tn_1osn0c
 e n1s0 
ns
[CPU0[]C[ PCPUwU12d]] t wdewtesdsttt_o_encsoet n1_0c onesn
c e1 01[ C0PnU 2sn] 
swd
test_once 10 ns
[CPU[0CP][U C1Pw]U2 d]w twddettesestts_o_tnco_e on10n cncsee
  1100 [Cn PsUn2
] swd
test_once 1[0 CnsP
U1] wd[tCeP[sCUtPU_02o]] nw cdtewe sdt1_to0ne cens s1t0
 _nso
nce [1CP0U 1n]s w
dtest_once 10 ns
[[CCPPUU01]]  wwddtteesstt__oonnccee  110000  nnss

[[C[CPPCUPUU021] ]] wwddtte esstwt__doonntcceee  s11t00_00  onnssn

ce 1[C0PU00[]C  PwUnd1t]se sw
td_toenscte_ o1n0c0e  n1s0
0 ns
[CP[UCP2U0][]C  PwUdwt1e]ds twtd_toenecsets _1otn0c0_e  nos1
0n0 cnse
 [1CP0U00][  CwPndUt1se]s 
tw_dotnecset _1o0n0c en s1
00 ns
[CP[UCP2U0][]C  PwUdwt1e]d swttd_toeensctse_ ot1n0c_0e  on1s0n
0 cnse
 [1C0PU00]  wdtnesst_o
nce 100 ns
[CPU0] wdtest_once 1000 ns
[CP[UC2[P]CPUU 0]1w w]ddt etswte_dsontctee_ 1so00t0n _ncso
en c1e0 [0CP10U00 ] nwdste
st_once 1000 ns
[CP[UCPU20]] wd tewst_dontce e10s00 tns_
once[C PU10] 0wd0te0st_ onnces 10
00 ns
[C[CPPU0]U wd2tes]t_o ncew 10d00 tns
est[C_PoUn0] cwdtees t_1onc0e 010000  nns
s
[CPU0] wdtest_onc[e C10P00U ns2
] w[CdPUt0]e wsdttes_t_ooncne c100e0  ns
1000[ CPUn0]s wd
test_once 1000 ns
[CPU2] wdtest_once 1000 ns
00 ns
[[CCPPUU2[0]C] P wUwd1dt]et sewtsd_tton_ecosent c_10eo0 n01c0 0en0 s01
00 0n0s0
 n[sCP
U2] w[dtCePstU_0o[]ncC ePw Ud11t00]e0 s0wt dn_tso
enscte_ o1n0[c0Ce0PU 021 ]0n 0swd0
t0e stn_son
ce [1C00P0U0 0n[]sC 
PwUd1t]e swt[dC_tPoUen2sc] tew_ do1tn0ec0set0 _01o 0nn0cs0e
 01 00n0s0 
ns
[CPU0][ CwPdUt[1eC]Ps Utw2_d] towendscttee_s ot1n_o0cn0ec0 e01 1 00n00s00
00  nsn
s
[CPU0][ CwPUd2t][ ewCsdPttUe_1so]tn_ cowend cte1e 0s100t00_000o  nnncsse

 10000 ns
[[CCPPU2U] 0wd]t[e sCtw_PodnUcte1 e1]0s0 0tw0_0do tnnse
cset _1[o0CnP0Uc02e]0  w0d1t e0snt0_so0n
c0e 010 00n00s n
s
[CPU0][ Cw[PdCUPtU12e]]s  wtdwt_edosttn_ecosnecte  _11o000n00c000e 0n s0
1 0n0s0
0[C0PU 2]n wsdt
est[_oCncPe U10000][0 0C wPndsU
t1e]s tw_[CdoPtUn2e]cs ewtd _t1eo0snt0_co0en0c 0e1  10n000s000
00 n0s
 ns
[CPU[0CP]U2 ] wwdd[tteCsetP_soUtnc1_e]o 1 n0w0c0d0e0t  en1ss
0t0_0o0n0[Cc PeUn2 s]1 
w0dt0es0t_0on0ce  1[n0C0s0P0
U0 0ns]
 wdtest_once 100000 ns
[C[PCPUU02]][  wCwdPdteUts1et]s_ otnw_cdeo tn1ec0s0e0t 0_10o00 n0nc0se
0 0100 0n0s[0
C0PU02]  nwdste
s[t_CoPncUe 01]0[ 00Cw0Pd0Ut01 e]nss 
tw_dotnecse[t C_P1Uo02n0] c0we0d 0te10s0 t0_n0osn0
c0e 01 00n00s0
0[ nCs
PU0][ CwPdUt[1eC]PsU t2w]_ dowtdnetcseestt __1oo0nnc0ce0 e01 0010000 000n00s0 n
0s
0 ns
[CPU[0CP]U2 ] wwddt[teCsePts_Uto1_n]coe n wc1d0e0t 0e10s000t0 _n0os0
n0c0e  ns1[
0C0PU02]0 0wd0te [snCts_Po
Unc0e] 1 00w0d00t[0e CnsPst
U_1o]n cwed t1e0s0t0_0o0n0c en s1
000000 ns
[CPU0] wd_start with max[i[CmPCuUmP2 U]d1 e]wlda _yws,dt a_crsattn awcriettl h w OimKta,hx  irmmeausxmti  md2ue1ml4 a7dy4e,8l 3ac6ya4,n6 c
ec[laC nPOcUKe0,l]   rOweKds,tt e rs2et1s_4tr7 e42c81u34r67s44i68v
3e[6 C41P60U
02[0]C0 P0wU0d1ut]se 
swtd_treesctu_rr[seCicPvuUer0 s]1i 0vw0ed0 _01s00t00au0rs0t
0 0wuist
h maximum d[eC[lPCaUPy2U,]1  ]cw adwn_dcs_etslat raOtrK t,w  iwrtiehts htm  am2xa1ix4mi7um4mu8 m3d 6ed4le6al
ya[,yC ,Pc Uac0na]cn ecwled ltO eKOs,Kt ,_r reresectsu tr2 s12i41v74e47 841380630460640
60[
0C[uPCsUP
2U]1 ]w dwtdetsets[_tCr_PerUce0uc]ru srwisdvi_evs et1 a01r00t00 00w00i00tu0hsu 
sm
aximum delay,[ C[cPCaUPn2Uc]1e ]lw  dwO_dKs_,ts atrarertst t w wi2it1th4h 7 m4ma8ax3xi6im4muum6m  
dde[elClaaPyyU,,0  ]cc aawnndccteeells  tOO_KKr,,e  crrueerssstti  v22e11 44177044088033066044066u

s[[
CCPPUU21]]  wwddtt[eeCssPttU__0rr]ee ccwuudrr_sssiitvvaeer  t11 00w00i00t00h00 00muuassx

imum delay, ca[n[CcCPePUlU2 1]O] K w,wd d_r_seststatar rt2t 1 w4wi7it4th8h 3 m6ma4ax6xi
im[muCumPm U d0de]el lawaydy,t, e cscatan_ncrceeelcl u OrOKsK,i, v rer ee1ss0tt0  02201104407u74s48
833664466

[[CCPP[UUC21P]]U  0ww]dd ttweedss_tts__trraeercctuu rrwssiiitvvhee   m11a00x00i00m00u00m00 uudsse

lay, cancel OK[,[C CPrPUeU2s1]t]   w2wd1d_4_s7st4ta8ar3rt6t 4 w6wi
it[thCh P mUma0ax]xi imwmudumtm e dsdetel_larayey,c, u crcasanincvceeel l 1 O0OK0K,0, 0 r0re0esustst 
 221144774488336[64C46P6
U
[0[C]CP PUwU2d1]_] s wwtddattreetss ttw__irrteehcc uumrrassxiiivvmeeu  m11 00d00e00l00a00y00,uu ssc

ancel OK, rest 2147483646
[CPU0] wdtest_recursive 1000000us
[CPU0] recursive wdog triggered 6 times, elapsed tick 12
[[C[PCUC0P]P UwUd2t1e]s]t _ rrerceeucrcsuiurvres s1ii0v0ve0e0 0 0ww0dudso
ogg  tt[rrCiPiUg0gg]g erererceuerdsdi v e6 6w  dottgi itmrmeigesgse,r, e de el6l aatpipmsesse,e dde l atptisiecdc kkt i c1k1 2212


[[[CCPCPU0PU]U 2w1]d]t e swwtd_drtteceeusrsstitv__er r1e0e0c0c0u0ur0r0sussii
vvee  [11C0P0U000]0 0r0e00c0u0r00si0uvues sw
d
og triggered 6 times, elapse[d[ CtCPiPcUkU2 11]2]
 [ rCrPeUe0cc]u uwrrdstseisitvv_eree  cwuwrddsoiovgge   1t0t0rr0i0i0g0g0gugse
erreedd[ C P6U60  ]t trieicmmueressisv,,e   wedelolga atpprsisgeegdedr e dt 6it ctiikmc ek1s ,2 1e
2la[
psC[ePdCU Pt2iUc]1k  ]1w2 
d[wtCPdeU0ts] etwds_tters_etr_cerueccruursrsiisvvei e1v0 0e01 000100000us00
0000000[uC0PsUu
0]s r
ecursive wdog [trCigPgeUre2d] [6  CtrPiemUecs1,u ]erl asprsieedv cteiuc rkw s1d2i
o[vCgePU  0t]w rwddiotgegsgt e_rtrecreuridsig veg6 1e 00rt00ei0d0m0 ues6
s ,t iem[ClePUas0]p, sr eecedulr saitpviesc wekdod g  1trt2iig
gce[kreC d P16 U2t2i
m]e[s ,C wePdlUatp1se]esd  ttw_icdrk te1e2c
s[uCtPrU_0s]ri ewvdcteeu str1_rs0eci0urv0sei0v e0 1100000000u00s000
u0s
00us
[[CPCUP0]U 2re]cu[ rsCrivPeeU cw1duo]rg  strireivgcegue rrewdsd 6io vtgiem e st,w redliaogpgsge ed trtriecikd g 1g26
e [rCtPeiUd0m]  e6wsd t,ets ite_mrleecausrps,isv ee e1d00 l0t0a0i0p0cussk
e d12 [
CtP[UiC0]cP rkUec 2ur]1s i2vwe
 dwdt[oegC strPitgUg_e1rre]de  6c wtuidmretss,e ieslvtaep_s erd1 e0tci0cuk0 r01s20i
0[vC0PeUu0 ]s1 w
0dt0es0t0_r0ec0ur0siuves 1
0000000us
[CPU0] recursive wdog tr[[iCCPgPUgU2e]1r ]er edrc ue6rc sutirvisemi ewvsdeo, g  wetdrloiaggpg setreredid g tg6ie ctrikem de1 s26,
  teliampesse,d  etliackp s1e2d
 tick[ CP12U[
0CP]U 2r] erceucurrs[siiCvvPeeU  1w]wd odrgeo cgtu rristgirgvieerg egwded ro6ge  dtt irm6ie gst,g ieemrleeaspd,s  e6ed l tatipimcsekes d,1 2 t
eilcakp s1ed2 
[tCPiUc2k]  1r2e
c[uCrPsUi0ve]  w[rdCeoPcgUu 1t]rr sirigegvceeru erwdsd i6ov get i tmwrediso,gg g eetlrraepisdge gde6 r teitcdik m 6e1 s2t,
i meelsa,p seelda[ pCtsPiUe2cd]k   rt1ei2ccu
kr si1v2e
 wdog t[rCigPgUe0r]ed  r6e tc[iuCmrePssU,i1 v]ee l arwpedsceoudgr  sttiirvciek g wg1d2eo
rg etdr i6gg [eCrtPeiUdm2 e]6 s rt,ei cmueerlss,ai pvesel eawdpd soetgdi  cttrkii cg1gke2 r
1e2d
 6 times[, CePlUa0[pC]sPe Udr1 ]et cirucerkc su1ri2sv
iev e wwddoo[ggC Pt Urt2i]rg giregergceuedr rs6ie vdtei  m6we sd,to igem letarspi,sg egdee rlteaidpc ks6 e 1d2t 
itmiecsk,  1e2la
psed tick 12
wdog_test end...