diff --git a/docs/2023/20231220_linux_console_tty.assets/7B94DDB4296F3A0016B229C6F1930890.png b/docs/2023/20231220_linux_console_tty.assets/7B94DDB4296F3A0016B229C6F1930890.png new file mode 100644 index 0000000..ca989a5 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/7B94DDB4296F3A0016B229C6F1930890.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/FD7909D83503F50DF2C479CF233E5447.png b/docs/2023/20231220_linux_console_tty.assets/FD7909D83503F50DF2C479CF233E5447.png new file mode 100644 index 0000000..4fc8c92 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/FD7909D83503F50DF2C479CF233E5447.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/IMG_20240105_103444_edit_409390889312531.jpg b/docs/2023/20231220_linux_console_tty.assets/IMG_20240105_103444_edit_409390889312531.jpg new file mode 100644 index 0000000..cbd2236 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/IMG_20240105_103444_edit_409390889312531.jpg differ diff --git a/docs/2023/20231220_linux_console_tty.assets/IMG_20240105_103510_edit_409420967453151.jpg b/docs/2023/20231220_linux_console_tty.assets/IMG_20240105_103510_edit_409420967453151.jpg new file mode 100644 index 0000000..5290ce7 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/IMG_20240105_103510_edit_409420967453151.jpg differ diff --git a/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_113843.png b/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_113843.png new file mode 100644 index 0000000..fa90163 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_113843.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_130738.png b/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_130738.png new file mode 100644 index 0000000..b0a3ce8 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_130738.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_150400.png b/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_150400.png new file mode 100644 index 0000000..f6cbb3e Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_150400.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_154656.png b/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_154656.png new file mode 100644 index 0000000..30a34c8 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_154656.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_155422.png b/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_155422.png new file mode 100644 index 0000000..83e1f2e Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/Screenshot_20240221_155422.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231227201119715.png b/docs/2023/20231220_linux_console_tty.assets/image-20231227201119715.png new file mode 100644 index 0000000..2906251 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231227201119715.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231227205831870.png b/docs/2023/20231220_linux_console_tty.assets/image-20231227205831870.png new file mode 100644 index 0000000..86c836c Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231227205831870.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231227205944602.png b/docs/2023/20231220_linux_console_tty.assets/image-20231227205944602.png new file mode 100644 index 0000000..45f59e8 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231227205944602.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231227210638335.png b/docs/2023/20231220_linux_console_tty.assets/image-20231227210638335.png new file mode 100644 index 0000000..5554a1d Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231227210638335.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231227210758837.png b/docs/2023/20231220_linux_console_tty.assets/image-20231227210758837.png new file mode 100644 index 0000000..bcd0506 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231227210758837.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231227211312722.png b/docs/2023/20231220_linux_console_tty.assets/image-20231227211312722.png new file mode 100644 index 0000000..092cb57 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231227211312722.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231227211353238.png b/docs/2023/20231220_linux_console_tty.assets/image-20231227211353238.png new file mode 100644 index 0000000..5b1b0f1 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231227211353238.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231227211710529.png b/docs/2023/20231220_linux_console_tty.assets/image-20231227211710529.png new file mode 100644 index 0000000..af4fcfa Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231227211710529.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231227211734840.png b/docs/2023/20231220_linux_console_tty.assets/image-20231227211734840.png new file mode 100644 index 0000000..52f3715 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231227211734840.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231227211935427.png b/docs/2023/20231220_linux_console_tty.assets/image-20231227211935427.png new file mode 100644 index 0000000..3ca7d87 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231227211935427.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229001347494.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229001347494.png new file mode 100644 index 0000000..bd02e73 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229001347494.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229001704973.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229001704973.png new file mode 100644 index 0000000..efbfe1d Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229001704973.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229002219506.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229002219506.png new file mode 100644 index 0000000..246097f Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229002219506.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229093152754.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229093152754.png new file mode 100644 index 0000000..0598de8 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229093152754.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229093329153.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229093329153.png new file mode 100644 index 0000000..bbb6043 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229093329153.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229095610270.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229095610270.png new file mode 100644 index 0000000..1bd9a43 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229095610270.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229101400585.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229101400585.png new file mode 100644 index 0000000..4fbd518 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229101400585.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229101637687.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229101637687.png new file mode 100644 index 0000000..ef5b3ea Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229101637687.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229101827871.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229101827871.png new file mode 100644 index 0000000..8871d67 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229101827871.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229102302019.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229102302019.png new file mode 100644 index 0000000..5964ca5 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229102302019.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229102333156.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229102333156.png new file mode 100644 index 0000000..386a593 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229102333156.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229102419643.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229102419643.png new file mode 100644 index 0000000..21bfae0 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229102419643.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229102704745.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229102704745.png new file mode 100644 index 0000000..2d92390 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229102704745.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229102902011.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229102902011.png new file mode 100644 index 0000000..56fab4c Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229102902011.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229103010300.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229103010300.png new file mode 100644 index 0000000..1f8a185 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229103010300.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229103201562.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229103201562.png new file mode 100644 index 0000000..7108192 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229103201562.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229104128936.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229104128936.png new file mode 100644 index 0000000..6e96982 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229104128936.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229104129932.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229104129932.png new file mode 100644 index 0000000..6e96982 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229104129932.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229104324874.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229104324874.png new file mode 100644 index 0000000..c1f1015 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229104324874.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229104711274.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229104711274.png new file mode 100644 index 0000000..287c7c7 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229104711274.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229105805483.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229105805483.png new file mode 100644 index 0000000..6b10722 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229105805483.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229110116743.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229110116743.png new file mode 100644 index 0000000..a42e97c Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229110116743.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229112758087.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229112758087.png new file mode 100644 index 0000000..841e5bf Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229112758087.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229113154976.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229113154976.png new file mode 100644 index 0000000..422fff4 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229113154976.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229113328873.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229113328873.png new file mode 100644 index 0000000..43fe4e4 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229113328873.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229113742293.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229113742293.png new file mode 100644 index 0000000..3e80969 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229113742293.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229113952968.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229113952968.png new file mode 100644 index 0000000..3373a27 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229113952968.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20231229114457559.png b/docs/2023/20231220_linux_console_tty.assets/image-20231229114457559.png new file mode 100644 index 0000000..ab0db38 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20231229114457559.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240104104844548.png b/docs/2023/20231220_linux_console_tty.assets/image-20240104104844548.png new file mode 100644 index 0000000..9104704 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240104104844548.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240104212642120.png b/docs/2023/20231220_linux_console_tty.assets/image-20240104212642120.png new file mode 100644 index 0000000..ef0e2be Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240104212642120.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240104212821894.png b/docs/2023/20231220_linux_console_tty.assets/image-20240104212821894.png new file mode 100644 index 0000000..f4d225c Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240104212821894.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240104213352370.png b/docs/2023/20231220_linux_console_tty.assets/image-20240104213352370.png new file mode 100644 index 0000000..52b9e1a Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240104213352370.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240104213612195.png b/docs/2023/20231220_linux_console_tty.assets/image-20240104213612195.png new file mode 100644 index 0000000..494a567 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240104213612195.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240104213635898.png b/docs/2023/20231220_linux_console_tty.assets/image-20240104213635898.png new file mode 100644 index 0000000..56e9c9c Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240104213635898.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240104214809328.png b/docs/2023/20231220_linux_console_tty.assets/image-20240104214809328.png new file mode 100644 index 0000000..5b9182f Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240104214809328.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240104222913453.png b/docs/2023/20231220_linux_console_tty.assets/image-20240104222913453.png new file mode 100644 index 0000000..3c0227b Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240104222913453.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240104235105066.png b/docs/2023/20231220_linux_console_tty.assets/image-20240104235105066.png new file mode 100644 index 0000000..eb195ff Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240104235105066.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240104235124096.png b/docs/2023/20231220_linux_console_tty.assets/image-20240104235124096.png new file mode 100644 index 0000000..8fc518a Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240104235124096.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240104235330145.png b/docs/2023/20231220_linux_console_tty.assets/image-20240104235330145.png new file mode 100644 index 0000000..99a1092 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240104235330145.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240104235435985.png b/docs/2023/20231220_linux_console_tty.assets/image-20240104235435985.png new file mode 100644 index 0000000..8ea38ce Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240104235435985.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240104235538121.png b/docs/2023/20231220_linux_console_tty.assets/image-20240104235538121.png new file mode 100644 index 0000000..5da4d15 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240104235538121.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105100903647.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105100903647.png new file mode 100644 index 0000000..4f0f4ee Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105100903647.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105104235288.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105104235288.png new file mode 100644 index 0000000..3956650 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105104235288.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105104450441.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105104450441.png new file mode 100644 index 0000000..8b015b4 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105104450441.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105104800259.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105104800259.png new file mode 100644 index 0000000..ff3230c Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105104800259.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105105030001.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105105030001.png new file mode 100644 index 0000000..eb20950 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105105030001.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105105109408.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105105109408.png new file mode 100644 index 0000000..2d96748 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105105109408.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105105558160.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105105558160.png new file mode 100644 index 0000000..b42f769 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105105558160.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105110055342.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105110055342.png new file mode 100644 index 0000000..ecd52ef Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105110055342.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105110818244.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105110818244.png new file mode 100644 index 0000000..4a288fd Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105110818244.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105110941938.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105110941938.png new file mode 100644 index 0000000..e374d0a Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105110941938.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105110953844.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105110953844.png new file mode 100644 index 0000000..0d3c67a Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105110953844.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105111057947.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105111057947.png new file mode 100644 index 0000000..3b996f2 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105111057947.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105111923043.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105111923043.png new file mode 100644 index 0000000..e2ab6ce Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105111923043.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105112222677.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105112222677.png new file mode 100644 index 0000000..24f3761 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105112222677.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105113429976.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105113429976.png new file mode 100644 index 0000000..04b5cbe Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105113429976.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105125854192.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105125854192.png new file mode 100644 index 0000000..62f8d64 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105125854192.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105130532207.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105130532207.png new file mode 100644 index 0000000..27230d2 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105130532207.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105130805622.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105130805622.png new file mode 100644 index 0000000..1a90d34 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105130805622.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105134324542.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105134324542.png new file mode 100644 index 0000000..415922f Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105134324542.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105141553967.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105141553967.png new file mode 100644 index 0000000..96d36b8 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105141553967.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105144114396.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105144114396.png new file mode 100644 index 0000000..52fbcff Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105144114396.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105144327093.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105144327093.png new file mode 100644 index 0000000..9dae0fd Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105144327093.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105144646017.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105144646017.png new file mode 100644 index 0000000..bfccae9 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105144646017.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105145112436.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105145112436.png new file mode 100644 index 0000000..e751659 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105145112436.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105145253177.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105145253177.png new file mode 100644 index 0000000..f837910 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105145253177.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105145319733.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105145319733.png new file mode 100644 index 0000000..f837910 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105145319733.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105145706138.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105145706138.png new file mode 100644 index 0000000..19f5932 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105145706138.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105145825115.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105145825115.png new file mode 100644 index 0000000..205a646 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105145825115.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240105154830009.png b/docs/2023/20231220_linux_console_tty.assets/image-20240105154830009.png new file mode 100644 index 0000000..f30b778 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240105154830009.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240108152509855.png b/docs/2023/20231220_linux_console_tty.assets/image-20240108152509855.png new file mode 100644 index 0000000..95be348 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240108152509855.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240108152544244.png b/docs/2023/20231220_linux_console_tty.assets/image-20240108152544244.png new file mode 100644 index 0000000..693ebd6 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240108152544244.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240108155108251.png b/docs/2023/20231220_linux_console_tty.assets/image-20240108155108251.png new file mode 100644 index 0000000..4726210 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240108155108251.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240108155256888.png b/docs/2023/20231220_linux_console_tty.assets/image-20240108155256888.png new file mode 100644 index 0000000..5e95a40 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240108155256888.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240108155310325.png b/docs/2023/20231220_linux_console_tty.assets/image-20240108155310325.png new file mode 100644 index 0000000..9a7adf2 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240108155310325.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240109114650351.png b/docs/2023/20231220_linux_console_tty.assets/image-20240109114650351.png new file mode 100644 index 0000000..95b5cd7 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240109114650351.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240112105209581.png b/docs/2023/20231220_linux_console_tty.assets/image-20240112105209581.png new file mode 100644 index 0000000..4ed822d Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240112105209581.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240112111356531.png b/docs/2023/20231220_linux_console_tty.assets/image-20240112111356531.png new file mode 100644 index 0000000..fe1df44 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240112111356531.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240112111800416.png b/docs/2023/20231220_linux_console_tty.assets/image-20240112111800416.png new file mode 100644 index 0000000..895c433 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240112111800416.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240112113426459.png b/docs/2023/20231220_linux_console_tty.assets/image-20240112113426459.png new file mode 100644 index 0000000..6100e8b Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240112113426459.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240112113558018.png b/docs/2023/20231220_linux_console_tty.assets/image-20240112113558018.png new file mode 100644 index 0000000..1a25f84 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240112113558018.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240112114347853.png b/docs/2023/20231220_linux_console_tty.assets/image-20240112114347853.png new file mode 100644 index 0000000..038745d Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240112114347853.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240112114359061.png b/docs/2023/20231220_linux_console_tty.assets/image-20240112114359061.png new file mode 100644 index 0000000..1dc71e4 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240112114359061.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240112115754979.png b/docs/2023/20231220_linux_console_tty.assets/image-20240112115754979.png new file mode 100644 index 0000000..6bd85f2 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240112115754979.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240119114538611.png b/docs/2023/20231220_linux_console_tty.assets/image-20240119114538611.png new file mode 100644 index 0000000..a688790 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240119114538611.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240119133807506.png b/docs/2023/20231220_linux_console_tty.assets/image-20240119133807506.png new file mode 100644 index 0000000..e0f11d7 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240119133807506.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240124170609711.png b/docs/2023/20231220_linux_console_tty.assets/image-20240124170609711.png new file mode 100644 index 0000000..38bd313 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240124170609711.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240124171207038.png b/docs/2023/20231220_linux_console_tty.assets/image-20240124171207038.png new file mode 100644 index 0000000..e6ef692 Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240124171207038.png differ diff --git a/docs/2023/20231220_linux_console_tty.assets/image-20240124220335289.png b/docs/2023/20231220_linux_console_tty.assets/image-20240124220335289.png new file mode 100644 index 0000000..5344fcf Binary files /dev/null and b/docs/2023/20231220_linux_console_tty.assets/image-20240124220335289.png differ diff --git a/docs/2023/20231220_linux_console_tty.md b/docs/2023/20231220_linux_console_tty.md index 17b3e5d..6a9311b 100644 --- a/docs/2023/20231220_linux_console_tty.md +++ b/docs/2023/20231220_linux_console_tty.md @@ -210,6 +210,14 @@ mknod console c 5 1 其中c代表character device, 5是主设备号，1是第一个子设备，这样就创建了/dev/console设备 +# 探究linux kernel启动流程 + +搞清楚aarch64下linux从entry开始到注册实际console的调用过程，以及linux kernel如何读取硬件dtb，并读出serial相关信息。 + +这里使用linux kernel 6.6.8的源码进行阅读 + +TODO + # NXP板子研究笔记 启动扳子，连接DEBUG到电脑，我这里默认连接到的串口tty为/dev/ttyACM0 @@ -222,7 +230,9 @@ mknod console c 5 1 默认启动的是自带的linux 5.4.70，通过查看启动拨码开关，默认是配置为eMMC启动。 -## NXP jailhouse +## NXP jailhouse初探 + +### 编译jailhouse https://github.com/nxp-imx/imx-jailhouse @@ -293,4 +303,1409 @@ cp tools/jailhouse export/jailhouse https://github.com/OK8MQ-linux-sdk/OK8MQ-linux-sdk -https://github.com/Comet959/-ok8mq-jailhouse-linux/releases/tag/toolchain +这个问题可以通过使用NXP提供的编译器以及环境来解决，从上面的URL中下载aarch64-poky-linux工具链并解压安装 + +![image-20231227201119715](20231220_linux_console_tty.assets/image-20231227201119715.png) + +```bash +Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g. +. /opt/fsl-imx-xwayland/5.4-zeus/environment-setup-aarch64-poky-linux +``` + +SDK源码包，包括linux kernel, 文件系统等 https://github.com/Comet959/-ok8mq-jailhouse-linux/releases/tag/v1.0.0 + +aarch64-poky-linux-gcc "stdio.h" not found https://blog.iotot.com/?p=346 + +由于之前是直接下载的linux-kernel build，那个build是使用aarch64-gnu-linux-gcc编译的，和这一套aarch64-poky工具链还是有区别的，所有我重新执行了 + +![image-20231227205831870](20231220_linux_console_tty.assets/image-20231227205831870.png) + +```bash +./ci/gen-kernel-build.sh +``` + +使用aarch64-poky工具链编译linux kernel 5.10 + +![image-20231227205944602](20231220_linux_console_tty.assets/image-20231227205944602.png) + +之后重新编译jailhouse，需要注意此时aarch64-poky的脚本依然生效，所以没有指定ARCH等变量 + +```bash +make KDIR=ci/out/linux/build-amd-seattle +``` + +![image-20231227210638335](20231220_linux_console_tty.assets/image-20231227210638335.png) + +依然找不到stdio.h，原因还是因为编译jailhouse的之后没有指定好sysroot，即使初始化脚本里设置了CC的sysroot，但是jailhouse编译的时候猜测并没有成功传入这个参数 + +![image-20231227210758837](20231220_linux_console_tty.assets/image-20231227210758837.png) + +检查发现hypervisor目录下的源代码都成功生成了object文件，而stdio.h是tools下的源码使用的，尝试修改tools/Makefile + +![image-20231227211312722](20231220_linux_console_tty.assets/image-20231227211312722.png) + +在KBUILD_CFLAGS中加一行，--sysroot指向aarch64-poky环境变量 + +![image-20231227211353238](20231220_linux_console_tty.assets/image-20231227211353238.png) + +成功编译了jailhouse可执行程序和jailhouse.ko，上板运行 + +![image-20231227211710529](20231220_linux_console_tty.assets/image-20231227211710529.png) + +jailhouse工具成功启动，但是添加内核模块的时候报错 invalid module format + +![image-20231227211734840](20231220_linux_console_tty.assets/image-20231227211734840.png) + +```bash +Linux OK8MP 5.4.70-2.3.0 #1 SMP PREEMPT Mon Apr 10 01:43:43 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux +``` + +刚才通过jailhouse的ci脚本编译的的是linux 5.10的内核，猜测是因为用的linux源码版本不对 + +![image-20231227211935427](20231220_linux_console_tty.assets/image-20231227211935427.png) + +模块加载时 insmod “Invalid module format ”问题解决 https://blog.csdn.net/ymangu666/article/details/22872439 + +> 内核无法加载模块的原因是因为记载版本号的字符串和当前正在运行的内核模块的不一样，这个版本印戳作为一个静态的字符串存在于内核模块中，叫vermagic + +而手头的NXP板子是5.4的系统。 + +接下来的目标就明确了，即在 https://github.com/Comet959/-ok8mq-jailhouse-linux/releases/tag/v1.0.0 下载厂家给的linux源码。 + +下载好之后放在opt目录，对应的linux-kernel源码目录位于`/opt/OK8MQ-linux-sdk/OK8MQ-linux-kernel` + +出现报错 + +```` +usr/bin/ld: scripts/dtc/dtc-parser.tab.o:(.bss+0x10): multiple definition of 'yylloc'; scripts/dtc/dtc-lexer.lex.o:(.bss+0x0): first defined here +```` + +查到的解决方法： + +``` +vim scripts/dtc/dtc-lexer.lex.c +640 + extern YYLTYPE yylloc; +``` + +在IMX的SDK下编写build.sh + +```bash +cd ./OK8MQ-linux-kernel +make -j16 imx_v8_defconfig +make -j16 +``` + +这样在jailhouse编译的时候使用: + +```bash +make KDIR=/opt/OK8MQ-linux-sdk/OK8MQ-linux-kernel +``` + +![image-20231229001347494](20231220_linux_console_tty.assets/image-20231229001347494.png) + +编译出来了面向linux 5.4内核的jailhouse.ko,上板试一下 + +![image-20231229001704973](20231220_linux_console_tty.assets/image-20231229001704973.png) + +还是报错,可以看到虽然都是5.4,但是和我目前板子运行的linux版本还是有区别的,注意到jailhouse本身也报了一个错是disagrees about version of symbol module_layout + +和当前板子上的kernel module对比: + +![image-20231229002219506](20231220_linux_console_tty.assets/image-20231229002219506.png) + +```bash +5.4.70-2.3.0 SMP preempt mod_unload modversions aarch64 # goodix +5.4.3 SMP preempt mod_unload modversions aarch64 # jailhouse +``` + +所以目前这份厂家提供的linux源码版本并不对,需要再向厂商要板子的linux 5.4.70源码. + +目前在向厂家要了，在29号拿到了下载权限 + +![image-20231229093152754](20231220_linux_console_tty.assets/image-20231229093152754.png) + +可以看到当前这版资料这里的linux内核和板子上的版本一致。 + +![image-20231229093329153](20231220_linux_console_tty.assets/image-20231229093329153.png) + +在下载这个版本的SDK后，我发现其根目录有一个build.sh + +![image-20231229095610270](20231220_linux_console_tty.assets/image-20231229095610270.png) + +试着用这个脚本编译IMX linux kernel 5.4.70 + +``` +. /opt/fsl-imx-xwayland/5.4-zeus/environment-setup-aarch64-poky-linux +./build.sh kernel +``` + +需要注意的是，我修改了build.sh，配置`MAKE_JOBS=8`，因为如果设置为16的话，会导致我的ubuntu直接死机，之前编译内核的之后没有出现过这种问题，所以我还是减少了并行编译的线程数。 + +![image-20231229101400585](20231220_linux_console_tty.assets/image-20231229101400585.png) + +我突然发现这个脚本甚至把jailhouse也编译了，也就算说这个SDK里是有jailhouse的源码的…… + +![image-20231229101637687](20231220_linux_console_tty.assets/image-20231229101637687.png) + +位于OK8MP-linux-kernel/extra/jailhouse，这下方便了许多，并且能够保证jailhouse.ko的版本和内核一致。 + +![image-20231229101827871](20231220_linux_console_tty.assets/image-20231229101827871.png) + +这里的vermagic终于对上了 + +![image-20231229102302019](20231220_linux_console_tty.assets/image-20231229102302019.png) + +### 上板运行jailhouse + +上板成功加载module + +![image-20231229102333156](20231220_linux_console_tty.assets/image-20231229102333156.png) + +/dev下会多一个jailhouse设备 + +![image-20231229102419643](20231220_linux_console_tty.assets/image-20231229102419643.png) + +接下来就是在板子上试一下jailhouse + +```bash +./jailhouse enable ../imx8mp.cell +``` + +![image-20231229102704745](20231220_linux_console_tty.assets/image-20231229102704745.png) + +成功配置了默认的板卡配置文件im8mp.cell，可以看到4个CPU都被识别到了。 + +```bash +./jailhouse cell create ../imx8mp-gic-demo.cell +``` + +![image-20231229102902011](20231220_linux_console_tty.assets/image-20231229102902011.png) + +开启gic-demo配置后，CPU3被关闭，查看cell list + +![image-20231229103010300](20231220_linux_console_tty.assets/image-20231229103010300.png) + +这里gic-demo被分配到了单独的CPU3上，还没有开始运行 + +```bash +./jailhouse cell start --name gic-demo +``` + +![image-20231229103201562](20231220_linux_console_tty.assets/image-20231229103201562.png) + +接下来试着启动一个non-root linux cell + +准备好kernel和ramdisk + +![image-20231229104324874](20231220_linux_console_tty.assets/image-20231229104324874.png) + + + +```bash +./tools/jailhouse cell linux ./imx8mp-linux-demo.cell ./kernel/Image -i ./kernel/ramdisk.img +``` + +![image-20231229104711274](20231220_linux_console_tty.assets/image-20231229104711274.png) + +看起来必须传入dtb文件 + +```bash +./tools/jailhouse cell linux ./imx8mp-linux-demo.cell ./kernel/Image --initrd ./kernel/ramdisk.img --dtb ./kernel/OK8MP-C.dtb +``` + +报错： + +![image-20231229105805483](20231220_linux_console_tty.assets/image-20231229105805483.png) + +![image-20231229110116743](20231220_linux_console_tty.assets/image-20231229110116743.png) + +可以看到是`fcntl.ioctl(self.dev, JailhouseCell.JAILHOUSE_CELL_CREATE, create)`出现了问题，报错OSError no.22 + +首先这段代码将self.dev指向了打开的/dev/jailhouse文件，之后执行了fcntl.ioctl，这里是执行了一个系统调用ioctl + +https://docs.python.org/zh-cn/3/library/fcntl.html + +https://manpages.debian.org/ioctl(2) + +> The **ioctl**() system call manipulates the underlying device parameters of special files. In particular, many operating characteristics of character special files (e.g., terminals) may be controlled with **ioctl**() requests. The argument *fd* must be an open file descriptor. + +fcntl.**ioctl**(*fd*, *request*, *arg=0*, *mutate_flag=True*) + +fd为文件描述符，request是一个termios参数（https://docs.python.org/zh-cn/3/library/termios.html#module-termios），arg指向一个buffer，ioctl可以理解为对一个device进行操作。怀疑是/dev/jailhouse没有权限写？ + +``` +chmod 777 /dev/jailhouse +``` + +依然报错`OSError: [Errno 22] Invalid argument` + +Error when testing linux on jailhouse https://groups.google.com/g/jailhouse-dev/c/WaSMOYB_XQ4/m/Kp62ClTpCgAJ + +https://www.mail-archive.com/jailhouse-dev@googlegroups.com/msg07264.html + +```bash +vi /usr/lib/python3.7/site-packages/pyjailhouse/cell.py +``` + +```bash +insmod jailhouse.ko +./tools/jailhouse enable ./imx8mp.cell +./tools/jailhouse cell linux ./imx8mp-linux-demo.cell ./kernel/Image --initrd ./kernel/ramdisk.img --dtb ./kernel/OK8MP-C.dtb +``` + +问题解决，应该首先使用enable启动jailhouse！ + +此时出现新的报错： + +![image-20231229112758087](20231220_linux_console_tty.assets/image-20231229112758087.png) + +可以看到linux-inmate-demo cell是成功创建了，但是： + +``` +FileNotFoundError: [Errno 2] No such file or directory: '/mnt/tools/../inmates/tools/arm64/linux-loader.bin' +``` + +jailhouse在/mnt目录下寻找`inmates/tools/arm64/linux-loader.bin`，但是我U盘里并没有这个文件。 + +然后板子就死机了： + +![image-20231229113154976](20231220_linux_console_tty.assets/image-20231229113154976.png) + +```bash +insmod jailhouse.ko +./tools/jailhouse enable ./imx8mp.cell +./tools/jailhouse cell linux \ + ./imx8mp-linux-demo.cell \ + ./kernel/Image \ + --initrd ./kernel/ramdisk.img \ + --dtb ./kernel/OK8MP-C.dtb +``` + +我在jailhouse源码里找到了这个目录 + +![image-20231229113328873](20231220_linux_console_tty.assets/image-20231229113328873.png) + +之后没有报错了，但是好像输出并没有打印出来： + +![image-20231229113742293](20231220_linux_console_tty.assets/image-20231229113742293.png) + +```bash +mount /dev/sda1 /mnt && cd /mnt +insmod jailhouse.ko +./tools/jailhouse enable ./imx8mp.cell +./tools/jailhouse cell linux \ + ./imx8mp-linux-demo.cell \ + ./kernel/Image \ + -i ./kernel/ramdisk.img \ + -d ./kernel/OK8MP-C.dtb \ + -c "console=ttymxc1,30890000,115200 earlycon=ttymxc1,0x30890000,115200" +``` + +查看一下板子的串口位置： + +![image-20231229113952968](20231220_linux_console_tty.assets/image-20231229113952968.png) + +![image-20231229114457559](20231220_linux_console_tty.assets/image-20231229114457559.png) + +可以看到板子默认使用的是ttymxc1，无论是console还是用户程序都往这个串口输出，通过USB线连接到我的ubuntu的/dev/ttyACM0外部tty。 + +non-root linux没输出的问题待解决！ + +## jailhouse cell探究 + +猜测出问题的一个原因是设备配置的问题，目前jailhouse的启动流程是，首先通过板卡的cell启动jailhouse，之后通过启动linux或自定义的cell以启动一个“虚拟器”并运行linux或裸机程序，而在启动non-root linux时，需要传递一个dtb文件，作为虚拟机linux启动的设备树，这些环节都需要进行研究和配置 + +在搜索资料的时候，发现了一个jailhouse资料仓库：https://github.com/CJTSAJ/jailhouse-learning + +### cell配置文件 + +`configs/arm64/imx8mp.c` + +```c +.header + .signature = JAILHOUSE_SYSTEM_SIGNATURE, + .hypervisor_memory = { + .phys_start = 0xfdc00000, + .size = 0x00400000, // 保留了4MB的内存供hypervisor使用 + }, + .debug_console = { + .address = 0x30890000, // 串口MMIO地址 + .size = 0x1000, // 串口寄存器MMIO区域大小 + .flags = JAILHOUSE_CON_ACCESS_MMIO | + JAILHOUSE_CON_REGDIST_4, + .type = JAILHOUSE_CON_TYPE_IMX, + }, + .root_cell = { + .name = "imx8mp", + + .num_pci_devices = ARRAY_SIZE(config.pci_devices), + .cpu_set_size = sizeof(config.cpus), // 子配置域cpus + .num_memory_regions = ARRAY_SIZE(config.mem_regions), // 子配置域mem_regions + .num_irqchips = ARRAY_SIZE(config.irqchips), // 子配置域irqchips + /* gpt5/4/3/2 not used by root cell */ + .vpci_irq_base = 51, /* Not include 32 base */ + }, + +.cpus = { + 0xf, // 4'b1111 +}, + +.mem_regions = { + ... // IVSHMEM 部分（inter vm shared memory) + // start at 0xfd9f.... + /* IO */ { + .phys_start = 0x00000000, + .virt_start = 0x00000000, + .size = 0x38800000, // 从0x0开始的984MB是IO区域 + .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | + JAILHOUSE_MEM_IO, + }, + /* RAM 00*/ { + .phys_start = 0x40000000, + .virt_start = 0x40000000, + .size = 0x80000000, // 从0x4000_0000开始的2G区域是RAM空间 + .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | + JAILHOUSE_MEM_EXECUTE, + }, + /* Inmate memory */{ // 这里配置好了虚拟机的内存区域 + // 从0xc0000000开始，分配了983MB的内存 + .phys_start = 0xc0000000, + .virt_start = 0xc0000000, + .size = 0x3d700000, + .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | + JAILHOUSE_MEM_EXECUTE, + }, + /* Loader */{ + .phys_start = 0xfdb00000, + .virt_start = 0xfdb00000, + .size = 0x100000, // 0xfdb00000存放loader + // 这里的loader指的是什么？查手册 + .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | + JAILHOUSE_MEM_EXECUTE, + }, + /* OP-TEE reserved memory?? */{ + .phys_start = 0xfe000000, + .virt_start = 0xfe000000, + .size = 0x2000000, + .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE, + }, + /* RAM04 */{ + .phys_start = 0x100000000, + .virt_start = 0x100000000, + .size = 0xC0000000, + // RAM04，从0x1_0000_0000开始，3G大小的内存空间 + .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE, + }, +}, + +.irqchips // 涉及GIC地址等信息 + /* GIC */ { + .address = 0x38800000, + .pin_base = 32, // 这里的pin指什么，查一下NXP的CPU手册 + .pin_bitmap = { + 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, + }, + }, + ... + +.pci_devices // 挂载了四个JAILHOUSE_PCI_TYPE_IVSHMEM设备 +``` + +接下来看一下linux-inmate的cell配置文件 + +`configs/arm64/imx8mp-linux-demo.c` + +```c +.cell = { + .signature = JAILHOUSE_CELL_DESC_SIGNATURE, // 不是SYSTEM配置而是CELL描述文件 + .revision = JAILHOUSE_CONFIG_REVISION, + .name = "linux-inmate-demo", + .flags = JAILHOUSE_CELL_PASSIVE_COMMREG, + + .cpu_set_size = sizeof(config.cpus), + .num_memory_regions = ARRAY_SIZE(config.mem_regions), + .num_irqchips = ARRAY_SIZE(config.irqchips), + .num_pci_devices = ARRAY_SIZE(config.pci_devices), + .vpci_irq_base = 154, /* Not include 32 base */ +}, + +.cpus = { + 0xc, // 4'b1100，也就是使用CPU0+1，之前启动这个cell的时候，CPU2和3被关闭了 +}, + +.mem_regions = { +/* UART2 earlycon */ { + .phys_start = 0x30890000, // 跟上面的debug_console一致 + .virt_start = 0x30890000, + .size = 0x1000, + .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | + JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED, + }, + /* UART4 */ { + .phys_start = 0x30a60000, // 加了一个新的UART地址，这个地址我记得是ttymxc0 + /* + [ 0.154619] 30860000.serial: ttymxc0 at MMIO 0x30860000 (irq = 28, base_baud = 5000000) is a IMX + [ 0.155042] 30880000.serial: ttymxc2 at MMIO 0x30880000 (irq = 29, base_baud = 5000000) is a IMX + [ 0.155376] 30890000.serial: ttymxc1 at MMIO 0x30890000 (irq = 30, base_baud = 1500000) is a IMX + [ 1.250079] printk: console [ttymxc1] enabled + */ + .virt_start = 0x30a60000, + .size = 0x1000, + .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | + JAILHOUSE_MEM_IO, + }, + /* SHDC3 */ { + // SD卡IO区域位于0x30b6_0000 + .phys_start = 0x30b60000, + .virt_start = 0x30b60000, + .size = 0x10000, + .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | + JAILHOUSE_MEM_IO, + }, + /* RAM: Top at 4GB Space */ { + .phys_start = 0xfdb00000, + .virt_start = 0, + .size = 0x10000, /* 64KB */ + .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | + JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE, + }, + /* RAM */ { + /* + * We could not use 0x80000000 which conflicts with + * COMM_REGION_BASE + + #define COMM_REGION_BASE 0x80000000 + + 位于include/inmate.h中定义，COMM REGION： + */ + /** Communication region between hypervisor and a cell. */ + // 这个内存区域用于hypervisor和cell进行通信 + struct jailhouse_comm_region { // 这里给出x86下的结构定义，ARM也有类似的定义 + COMM_REGION_GENERIC_HEADER; + /** I/O port address of the PM timer (x86-specific). */ + __u16 pm_timer_address; + /** Number of CPUs available to the cell (x86-specific). */ + __u16 num_cpus; + /** Calibrated TSC frequency in kHz (x86-specific). */ + __u32 tsc_khz; + /** Calibrated APIC timer frequency in kHz or 0 if TSC deadline timer + * is available (x86-specific). */ + __u32 apic_khz; + } __attribute__((packed)); + + #define COMM_REGION_GENERIC_HEADER \ + /** Communication region magic JHCOMM */ \ + char signature[6]; \ + /** Communication region ABI revision */ \ + __u16 revision; \ + /** Cell state, initialized by hypervisor, updated by cell. */ \ + volatile __u32 cell_state; \ + /** Message code sent from hypervisor to cell. */ \ + volatile __u32 msg_to_cell; \ + /** Reply code sent from cell to hypervisor. */ \ + volatile __u32 reply_from_cell; \ + /** Holds static flags, see JAILHOUSE_COMM_FLAG_*. */ \ + __u32 flags; \ + /** Debug console that may be accessed by the inmate. */ \ + struct jailhouse_console console; \ + /** Base address of PCI memory mapped config. */ \ + __u64 pci_mmconfig_base; + + /* + + */ + .phys_start = 0xc0000000, + .virt_start = 0xc0000000, + .size = 0x3d700000, // 983MB内存，从0xc000_0000物理地址开始 + .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | + JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_DMA | + JAILHOUSE_MEM_LOADABLE, + }, + /* communication region */ { + .virt_start = 0x80000000, // 上面提到的通信区域 + .size = 0x00001000, // 4KB + .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | + JAILHOUSE_MEM_COMM_REGION, + }, +``` + +### i.MX8板子的dts文件 + +![image-20240104212642120](20231220_linux_console_tty.assets/image-20240104212642120.png) + +位于厂家SDK中的`{linux kernel src}/arch/arm64/boot/dts/freescale/OK8MP-C.dts` + +![image-20240104212821894](20231220_linux_console_tty.assets/image-20240104212821894.png) + +依赖头文件`imx8mp.dtsi`和`pd.h`，正如之前学长所讲的各级厂商针对自己的板子对上游dts文件进行改造。 + +在OK8MP-C的dts中有一个`/chosen/stdout-path`节点指向了uart2： + +![image-20240104213352370](20231220_linux_console_tty.assets/image-20240104213352370.png) + +https://community.nxp.com/t5/i-MX-Processors/Specify-console-in-device-tree-chosen-stdout-path/td-p/1286547 + +一般也可以写成`stdout-path = "/serial@{addr}:115200"`，这里写uart2是因为头文件里有定义： + +![image-20240104213612195](20231220_linux_console_tty.assets/image-20240104213612195.png) + +![image-20240104213635898](20231220_linux_console_tty.assets/image-20240104213635898.png) + +根据NXP提供的i.MX手册： + +![image-20240104214809328](20231220_linux_console_tty.assets/image-20240104214809328.png) + +这里就和uboot启动linux时的： + +``` +[ 0.155376] 30890000.serial: ttymxc1 at MMIO 0x30890000 (irq = 30, base_baud = 1500000) is a IMX +[ 1.250079] printk: console [ttymxc1] enabled +``` + +对应上了，并且dtb中定义的3个UART都被linux识别出来了 + +接下来先看OK8MP-C的设备树 + +```bash +/reserved-memory + /vdev0vring0 + /vdev0vring1 + /vdevbuffer + /rsc-table + /rpmsg_reserved + /imx8mp-cm7 + +/chosen + /stdout-path=&uart2 + +/leds # 灯 +/keys # 按键 + +/reg_usb1_host_vbus +/reg_usdhc2_vmmc +/usdhc1_pwrseq +/reg_audio_pwr # 音频 +/cbtl04gp # 交叉开关crossbar +/bt_sco_codec # 编码器 +/sound-bt-sco # 声卡 +/sound-hdmi # HDMI +/sound-wm8960 # wm8960声卡 +/sound-nau8822 # nau8822立体声音频解码器 +/sound-xcvr # 收发器模组 +/lvds_backlight # 低振幅差分信号背光 +/dsi_backlight # dsi LCD 背光 + +# 下面是设备树追加数据，对已存在的节点追加定义 + +&aud2htx { # Audio Subsystem TO HDMI TX Subsystem + status = "okay"; +}; + +&clk { # 时钟 + init-on-array = ; +}; + +&A53_0 # CPU +... +&i2c* # I2C +... +&pcie +... +&uart1 { /* BT */ + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_uart1>; + assigned-clocks = <&clk IMX8MP_CLK_UART1>; + assigned-clock-parents = <&clk IMX8MP_SYS_PLL1_80M>; + fsl,uart-has-rtscts; + status = "okay"; +}; +&uart2 { + /* console */ + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_uart2>; + status = "okay"; +}; +&uart3 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_uart3>; + assigned-clocks = <&clk IMX8MP_CLK_UART3>; + assigned-clock-parents = <&clk IMX8MP_SYS_PLL1_80M>; + fsl,uart-has-rtscts; + status = "okay"; +}; +... +&wdog1 # watchdog +&iomuxc # pin 控制 + /pinctrl_uart* +... +&gpu_3d { + status = "okay"; +}; +&gpu_2d { + status = "okay"; +}; +``` + +![image-20240104222913453](20231220_linux_console_tty.assets/image-20240104222913453.png) + +**AXI** (Advanced eXtensible Interface) 高级可拓展接口 + +**AHB** (Advanced High-performance Bus) 高级高性能总线 + +**SPBA** (Shared Peripheral Bus Arbiter) + +**SDMA** (Smart Direct Memory Access) + +**AP Peripherals** (Application Processor Peripherals) ? + +## 修改dts并编译 + +https://zhuanlan.zhihu.com/p/656691650 + +需要在厂家提供的linux源码里添加修改后的dts文件，位于`arch/arm64/boot/dts` + +```bash +cd /opt/nxp/OK8MP-linux-sdk/OK8MP-linux-kernel +. /opt/fsl-imx-xwayland/5.4-zeus/environment-setup-aarch64-poky-linux +make ARCH=arm64 -j8 CROSS_COMPILE=aarch64-poky-linux- dtbs # += Image for build image +``` + +![image-20240104235105066](20231220_linux_console_tty.assets/image-20240104235105066.png) + +![image-20240104235124096](20231220_linux_console_tty.assets/image-20240104235124096.png) + +修改Makefile，添加`OK8MP-C-wheatfox.dtb` + +![image-20240104235330145](20231220_linux_console_tty.assets/image-20240104235330145.png) + +![image-20240104235435985](20231220_linux_console_tty.assets/image-20240104235435985.png) + +虽然有报错，但实际上dtc只是报出了warning，不过makefile默认认为失败了，可以看到新的dtb文件也生成了： + +![image-20240104235538121](20231220_linux_console_tty.assets/image-20240104235538121.png) + +接下来修改扳子启动时的dtb + +`OK8MP-C-board.dts` + +只留下默认的那个uart2（serial1, ttymxc1） + +![image-20240105104235288](20231220_linux_console_tty.assets/image-20240105104235288.png) + +编译后放到u盘里 + +![image-20240105104450441](20231220_linux_console_tty.assets/image-20240105104450441.png) + +## 修改imx8mp板子启动的设备树 + +打电话问forlinx技术，了解到进入`/run/media/mmcblk2p`，对应eMMC存储，这个里面放着Image等启动文件 + +![image-20240105100903647](20231220_linux_console_tty.assets/image-20240105100903647.png) + +修改启动时使用的dtb，只需要替换这里的`OK8MP-C.dtb`即可，注意名字一定保持不变，否则uboot无法找到设备树文件。 + +接下来将刚刚修改好的板子启动dtb对原dtb进行替换。 + +```bash +cp OK8MP-C-board.dtb /run/media/mmcblk2p1/OK8MP-C.dtb +``` + +![image-20240105104800259](20231220_linux_console_tty.assets/image-20240105104800259.png) + +重启后发现设备还是被识别出来了，因为之前只是注释了&节点追加定义，如果要彻底隐藏节点，推测需要在头文件里把根下的节点去掉才可以！ + +所以需要修改`imx8mp.dtsi` ？ + +![image-20240105105030001](20231220_linux_console_tty.assets/image-20240105105030001.png) + +![image-20240105105109408](20231220_linux_console_tty.assets/image-20240105105109408.png) + +直接修改头文件会导致大量的dts编译报错，因为很多dts需要这些uart1和uart3节点： + +![image-20240105105558160](20231220_linux_console_tty.assets/image-20240105105558160.png) + +试着在OK8MP厂家dts添加节点补充属性status = "disabled"？ + +```c +&uart1 { + status = "disabled"; +}; + +&uart3 { + status = "disabled"; +}; + +&uart2 { + /* console */ + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_uart2>; + status = "okay"; +}; +``` + +成功！更换dtb并重启后，**linux只初始化了uart2(serial1,ttymxc1)**： + +![image-20240105110055342](20231220_linux_console_tty.assets/image-20240105110055342.png) + +可以看到uart1和uart3都没有被linux识别。 + +## 启动linux inmate研究 + +我注意到，厂家给的dts里是有linux inmate系列的dts的： + +![image-20240105111923043](20231220_linux_console_tty.assets/image-20240105111923043.png) + +复制一份进行修改：`arch/arm64/boot/dts/freescale/imx8mp-evk-inmate-wheatfox.dts` + +![image-20240105112222677](20231220_linux_console_tty.assets/image-20240105112222677.png) + +发现这个inmate dts使用的是uart4（恰好和板子定义的前三个串口没有冲突），先试着用这个默认的设备树启动inmate，串口暂时还是用serial1 + +```bash +./tools/jailhouse enable ./imx8mp.cell +./tools/jailhouse cell linux \ + ./imx8mp-linux-demo.cell \ + ./kernel/Image \ + -i ./kernel/ramdisk.img \ + -d ./kernel/imx8mp-evk-inmate-wheatfox.dtb \ + -c "clk_ignore_unused console=ttymxc1,0x30890000,115200 earlycon=ec_imx6q,0x30890000,115200" +``` + +![image-20240105113429976](20231220_linux_console_tty.assets/image-20240105113429976.png) + +理论上应该要看到Started cell "linux-inmate-demo"才说明成功启动 + +尝试修改jailhouse源码，在cell_start函数添加printk输出调试信息 + +![image-20240105125854192](20231220_linux_console_tty.assets/image-20240105125854192.png) + +从`cell_set_loadable`函数开始排查。 + +![image-20240105130532207](20231220_linux_console_tty.assets/image-20240105130532207.png) + +这里涉及到jailhouse对hypercall的处理，也就是说，cell_start是通过一个hypercall进行函数执行的。 + +在cell.c中，jailhouse_cmd_cell_start执行了hypercall： + +![image-20240105130805622](20231220_linux_console_tty.assets/image-20240105130805622.png) + +可以看到在cell_load之后，jailhouse cmd又发送了一个cell_load的hypercall。 + +![image-20240105141553967](20231220_linux_console_tty.assets/image-20240105141553967.png) + +编译的时候出现了问题，在修改了hypervisor firmware部分的代码后，kernel module对应部分并没有更新，及时代码已经被修改了。 + +在driver/main.c中制定了MODULE_FIRMWARE，在我手动修改为jailhouse.bin1，后，上板运行出现了报错： + +![image-20240105144114396](20231220_linux_console_tty.assets/image-20240105144114396.png) + +说明kernel module和其注册的firmware（这里即为jailhouse baremetal部分的程序）是分离的，我刚刚修改了firmware代码后，并没有更新在板子上。 + +![image-20240105144327093](20231220_linux_console_tty.assets/image-20240105144327093.png) + +request_firware第一个参数用来保存读取的firmware，第二个参数是固件文件名，第三个是申请固件的设备。 + +那么固件应该放在哪里呢： + +![image-20240105145112436](20231220_linux_console_tty.assets/image-20240105145112436.png) + +果然在板子的lib/firmware里找到了一直没被更新的固件jailhouse.bin： + +![image-20240105145319733](20231220_linux_console_tty.assets/image-20240105145319733.png) + +```bash +cp jailhouse.bin /lib/firmware/jailhouse.bin +``` + +成功更新jailhouse firmware： + +![image-20240105145706138](20231220_linux_console_tty.assets/image-20240105145706138.png) + +printk排查问题后，发现python脚本在load linux kernel的时候就卡住了，甚至没有开始执行load dtb和load initrd，如何解决？ + +![image-20240105154830009](20231220_linux_console_tty.assets/image-20240105154830009.png) + +继续打print输出，最后定位到`driver/cell.c:load_image`函数： + +![image-20240108152509855](20231220_linux_console_tty.assets/image-20240108152509855.png) + +![image-20240108152544244](20231220_linux_console_tty.assets/image-20240108152544244.png) + +可以看到，卡在了： + +```c +if (copy_from_user(image_mem + page_offs, + (void __user *)(unsigned long)image.source_address, + image.size)) + err = -EFAULT; +``` + +导致后面的printk并没有输出，可以确定的是image传递是成功的，并且也找到了放image的内存区域`0xc000_0000`，image加载的物理地址为`0xc0280000`，image大小为`0x1aea200`，为`26.91455078125MB`，这个和传入的文件大小吻合。 + +也就是说目前整个inmate linux启动程序在`load_image`中的`copy_from_user`函数卡住了！ + +查了一下发现，copy_from_user并不是jailhouse定义的函数，而是linux kernel提供的函数。 + +(linux 6.7) + +![image-20240108155310325](20231220_linux_console_tty.assets/image-20240108155310325.png) + +![image-20240108155256888](20231220_linux_console_tty.assets/image-20240108155256888.png) + + + +其中to是需要加载到的目标内核空间地址，from则是用户空间中的源地址，n表示需要copy多少字节。 + +### OK8MP硬件信息梳理 + +`CPU: i.MX8MP[8] rev1.1 1600 MHz (running at 1200 MHz)` + +*i.MX 8M Plus Applications Processor Reference Manual Document Number: IMX8MPRM Rev. 1, 06/2021* + +执行cat /proc/iomem后，涉及RAM的部分： + +``` +40000000-53ffffff : System RAM + 40480000-41b5ffff : Kernel code + 41b60000-41e2ffff : reserved + 41e30000-4206bfff : Kernel data + 43000000-43010fff : reserved +54ff0000-54ffffff : System RAM +55010000-550fefff : System RAM +55100000-553fffff : System RAM +55500000-557fffff : System RAM +58000000-923fffff : System RAM + 62000000-91ffffff : reserved +94400000-bfffffff : System RAM + bd600000-bd9fffff : reserved + bdbfe000-bf9fffff : reserved + bfa72000-bfb72fff : reserved + bfb73000-bfbd2fff : reserved + bfbd5000-bfbdafff : reserved + bfbdb000-bfbdbfff : reserved + bfbdc000-bfffffff : reserved +``` + +结合imx8mp.cell： + +```c +IO: 0x00000000 - 0x40000000 size=0x40000000 1G +RAM01: 0x40000000 - 0xc0000000 size=0x80000000 2G +Inmate RAM: 0xc0000000 - 0xfd700000 size=0x3d700000 983M +Loader: 0xfdb00000 - 0xfdc00000 size=0x100000 1M +... +RAM04: 0x100000000 - 0x1c0000000 size=0xC0000000 3G +``` + +imx8mq-linux-demo.cell： + +```c +UART1_ec: 0x30860000 +UART2: 0x30890000 +SHDC1: 0x30b40000 +RAM: 0xc0000000 - 0xfdc00000 size=0x3dc00000 988M +Comm: vaddr 0x80000000 +``` + +这里我发现了一个问题，板子上打印出来的iomem中的物理内存区域和cell文件里的对不上，以及Image的load_addr： + +```c +[wheatfox|python] trying to load kernel into cell, addr=0xc0280000 +``` + +这个地址是保存在哪里的？如果0xc这个地址并没有在板子上被映射到RAM，那为什么编译出来的Image要加载到这里？还有一个Loader区域，这个区域是能够正常加载linux-loader.bin的，但是0xfdb这个地址也没有在iomem中显示。 + +接下来启动jailhouse，然后再打印一下iomem： + +```bash +... +3d800000-3dbfffff : 3d800000.ddr_pmu ddr_pmu@3d800000 +# END OF IO MEM +40000000-53ffffff : System RAM + 40480000-41b5ffff : Kernel code + 41b60000-41e2ffff : reserved + 41e30000-4206bfff : Kernel data + 43000000-43010fff : reserved +54ff0000-54ffffff : System RAM +55010000-550fefff : System RAM +55100000-553fffff : System RAM +55500000-557fffff : System RAM +58000000-923fffff : System RAM + 62000000-91ffffff : reserved +94400000-bfffffff : System RAM + bd600000-bd9fffff : reserved + bdbfe000-bf9fffff : reserved + bfa72000-bfb72fff : reserved + bfb73000-bfbd2fff : reserved + bfbd5000-bfbdafff : reserved + bfbdb000-bfbdbfff : reserved + bfbdc000-bfffffff : reserved +# 下面这部分是jailhouse新建的映射 +fd700000-fd7fffff : PCI ECAM +fd800000-fd803fff : pci@0 + fd800000-fd800fff : 0000:00:00.0 + fd800000-fd800fff : uio_ivshmem[0000:00:00.0] + fd801000-fd801fff : 0000:00:01.0 + fd801000-fd801fff : ivshmem-net +fd900000-fd900fff : uio_ivshmem[0000:00:00.0] +fd901000-fd909fff : uio_ivshmem[0000:00:00.0] +fd90a000-fd90ffff : uio_ivshmem[0000:00:00.0] +fda00000-fda00fff : ivshmem-net +fda01000-fdafefff : ivshmem-net +fdc00000-fdffffff : Jailhouse hypervisor +``` + +虽然启动jailhouse后有一些新的mem区域，但是里面也没有0xc0000000和0xfdb00000。 + +### 调整linux inmate内存配置 + +查一下jailhouse的python脚本是如何获得image的load addr的，其核心逻辑位于`ARMCommon.setup()`中： + +![image-20240109114650351](20231220_linux_console_tty.assets/image-20240109114650351.png) + +这里的逻辑是，如果找到一个内存区域，则load会自动设置为这个内存区域的开头，然后dtb等附加文件依次摆放。 + +我修改了imx8mp-linux-demo.c的配置，将0xc开头的内存区域调整为0x6000_0000，大小暂定为256MB，因为我目前的板子是2G内存的型号，而默认的cell文件似乎是为4G内存准备的，0xc开头的部分在我的板子上并不存在。调整后成功加载并启动linux inmate： + +![image-20240112105209581](20231220_linux_console_tty.assets/image-20240112105209581.png) + +可以看到内存区域确实是限制在0x6000这里的配置好的区域了： + +```c +[ 0.000000] NUMA: Faking a node at [mem 0x0000000060000000-0x000000006fffffff] +[ 0.000000] NUMA: NODE_DATA [mem 0x6ff7c500-0x6ff7dfff] +[ 0.000000] Zone ranges: +[ 0.000000] DMA32 [mem 0x0000000060000000-0x000000006fffffff] +[ 0.000000] Normal empty +[ 0.000000] Movable zone start for each node +[ 0.000000] Early memory node ranges +[ 0.000000] node 0: [mem 0x0000000060000000-0x000000006fffffff] +[ 0.000000] Initmem setup node 0 [mem 0x0000000060000000-0x000000006fffffff] +``` + +并且启动的CPU也确实只有两个： + +```c +[ 0.000000] Booting Linux on physical CPU 0x0000000002 [0x410fd034] +... +[ 0.063542] CPU1: Booted secondary processor 0x0000000003 [0x410fd034] +[ 0.063602] smp: Brought up 1 node, 2 CPUs +[ 0.083838] SMP: Total of 2 processors activated. +[ 0.088338] CPU features: detected: 32-bit EL0 Support +[ 0.093262] CPU features: detected: CRC32 instructions +``` + +启动时遇到的第一个严重的问题是ramdisk“损坏”： + +```c +[ 0.442452] Unpacking initramfs... +[ 0.445603] Initramfs unpacking failed: invalid magic at start of compressed archive +[ 0.455219] Freeing initrd memory: 8696K +... +[ 1.292349] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) +``` + +因为我传递的不是cpio格式的ramfs而是一个ramdisk.img，猜测可能是这里的原因，那么就需要手动编译一个loongarch64的rootfs并打包为cpio + +### 编译一个rootfs + +下载busybox 1.36.0源码。 + +```bash +. /opt/fsl-imx-xwayland/5.4-zeus/environment-setup-aarch64-poky-linux +make ARCH=arm64 CROSS_COMPILE=aarch64-poky-linux- menuconfig +make ARCH=arm64 CROSS_COMPILE=aarch64-poky-linux- -j16 +make ARCH=arm64 CROSS_COMPILE=aarch64-poky-linux- install CONFIG_PREFIX=./build +``` + +![image-20240112111356531](20231220_linux_console_tty.assets/image-20240112111356531.png) + +https://community.nxp.com/t5/Layerscape/busybox-compile/td-p/735970 + +实际上在厂家提供的SDK里就有编译rootfs和ramdisk的部分，可以研究一下： + +![image-20240112111800416](20231220_linux_console_tty.assets/image-20240112111800416.png) + +```bash +#!/bin/bash + +dd if=/dev/zero of=$SDK_PATH/images/ramdisk.ext4 bs=1M count=0 seek=32 +mkfs.ext4 -F -i 4096 $SDK_PATH/images/ramdisk.ext4 -d $SDK_PATH/tools/ramdisk +gzip --best -c $SDK_PATH/images/ramdisk.ext4 > $SDK_PATH/images/ramdisk.ext4.gz + +mkimage -n "ramdisk" -A arm -O linux -T ramdisk -C gzip -d $SDK_PATH/images/ramdisk.ext4.gz $SDK_PATH/images/ramdisk.img +rm $SDK_PATH/images/ramdisk.ext4 $SDK_PATH/images/ramdisk.ext4.gz +#pushd $SDK_PATH/tools/ramdisk/ +#find . | cpio -o -Hnewc | gzip -9 > $SDK_PATH/tools/ramdisk.cpio.gz +#popd + +#$SDK_PATH/tools/bin/mkimage -n "ramdisk" -A arm -O linux -T ramdisk -C none -d $SDK_PATH/tools/ramdisk.cpio.gz $SDK_PA#TH/images/ramdisk.img + +#rm $SDK_PATH/tools/ramdisk.cpio.gz +``` + +可以看到，这里的img实际上是uboot image镜像，通过把ramdisk.ext4文件mkimage制作而成，接下来需要尝试制作cpio文件。 + +因为之前在启动rust-shyper的时候，我编译了一份arm64的busybox的rootfs，试着打包为cpio： + +```bash +find ./build | cpio -o -H newc | gzip > rootfs.cpio.gz +``` + +更换cpio后没有unpacking报错了，但是出现了新的unhandled trap： + +``` +FATAL: unhandled trap (exception class 0x24) +Cell state before exception: + pc: ffff800010a5dbd8 lr: ffff800010a50f38 spsr: 20000005 EL1 + sp: ffff800011c93bd0 esr: 24 1 1830006 ESR_RAW=0000000093830006 + x0: ffff000078818110 x1: ffff00007881ab80 x2: ffff800011f581f0 + x3: ffff800010a5dbd8 x4: 0000000000000000 x5: ffff80006e349000 + x6: 0000000002dbd7d7 x7: ffff00007fb80540 x8: ffff000015097a20 + x9: ffff800011c93d50 x10: 00000000000009c0 x11: 0000000000000166 +x12: 0000000000000030 x13: 0000000000000000 x14: ffffffffffffffff +x15: ffff0000151f4a70 x16: 0000000000000000 x17: 0000000000000000 +x18: 0000000000000010 x19: 0000000000000000 x20: 0000000000000000 +x21: ffff000078818840 x22: ffff800011f581f0 x23: 00000000000023e0 +x24: ffff0000771ee3e0 x25: 000000000000012c x26: ffff0000771ec000 +x27: ffff00007881ab80 x28: ffff000078818840 x29: ffff800011c93bd0 +``` + +这里的FATAL: unhandled trap (exception class 0x24)是jailhouse的输出： + +```c +void arch_handle_trap(union registers *guest_regs) +{ + struct trap_context ctx; + trap_handler handler; + int ret = TRAP_UNHANDLED; + + fill_trap_context(&ctx, guest_regs); + + handler = trap_handlers[ESR_EC(ctx.esr)]; + if (handler) + ret = handler(&ctx); + + if (ret == TRAP_UNHANDLED || ret == TRAP_FORBIDDEN) { + panic_printk("\nFATAL: %s (exception class 0x%02llx)\n", + (ret == TRAP_UNHANDLED ? "unhandled trap" : + "forbidden access"), + ESR_EC(ctx.esr)); + panic_printk("Cell state before exception:\n"); + dump_regs(&ctx); + panic_park(); + } +} +``` + +调查一下ESR的内容：`0000000093830006` + +https://esr.arm64.dev/#2474835974 + +![image-20240112115754979](20231220_linux_console_tty.assets/image-20240112115754979.png) + +``` +EC: 0x24 0b100100 +Data Abort from a lower Exception level +32-bit instruction trapped +Abort caused by reading from memory +Translation fault, level 2. +``` + +## 调试non-root linux + +### jailhouse flags研究 + +jailhouse中对内存属性的配置flag位于`include/jailhouse/cell-config.h` + +```c +#define JAILHOUSE_MEM_READ 0x0001 // 内存可读 +#define JAILHOUSE_MEM_WRITE 0x0002 // 内存可写 +#define JAILHOUSE_MEM_EXECUTE 0x0004 // 内存可执行 +#define JAILHOUSE_MEM_DMA 0x0008 // 用于外设进行DMA的区域 +#define JAILHOUSE_MEM_IO 0x0010 // IO区域 +#define JAILHOUSE_MEM_COMM_REGION 0x0020 // jailhouse communication区域 +#define JAILHOUSE_MEM_LOADABLE 0x0040 // 可加载（image, loader, dtb, initramfs...） +#define JAILHOUSE_MEM_ROOTSHARED 0x0080 // root和non-root的共享内存区域 +#define JAILHOUSE_MEM_NO_HUGEPAGES 0x0100 // 该区域的映射不使用大页 +#define JAILHOUSE_MEM_IO_UNALIGNED 0x8000 // IO，访问的地址需要在access data size上对齐，若未对齐则报错，除非mem配置了这个flag +#define JAILHOUSE_MEM_IO_WIDTH_SHIFT 16 /* uses bits 16..19 */ +#define JAILHOUSE_MEM_IO_8 (1 << JAILHOUSE_MEM_IO_WIDTH_SHIFT) // 下面4个在jailhouse源码里并没有处理的代码 +#define JAILHOUSE_MEM_IO_16 (2 << JAILHOUSE_MEM_IO_WIDTH_SHIFT) // 其主要通过size< **物理中断**是响应处理器外部信号而生成的中断。通常由外围设备生成。系统不会让核心不断轮询外部信号，而是通过生成中断来通知核心需要进行某些操作。 +> +> 例如，一个系统可能使用通用异步收发器/发送器（UART）接口与外部世界进行通信。当UART接收到数据时，它需要一种机制来告诉处理器新数据已经到达并且可以被处理。UART可以使用的一种机制是生成中断来向处理器发出信号。 +> +> 复杂的系统可能具有许多不同优先级的中断源，包括嵌套中断处理的能力，其中更高优先级的中断可以中断较低优先级的中断。处理器对这些事件的响应速度可能是系统设计中的关键问题，称为中断延迟。接下来，我们将看一下不同类型的物理中断。 +> +> **SError** +> +> 系统错误（SError）是一种异常类型，旨在由内存系统响应意外事件而生成。我们不期望这些事件，但需要知道它们是否发生。这些是异步报告的，因为触发事件的指令可能已经被执行。 +> +> SError的一个典型示例是以前称为外部异步中止的情况。SError中断的示例包括： +> +> 1. 内存访问已通过所有MMU检查，但在内存总线上遇到错误 +> 2. 一些RAM的奇偶校验或纠错码（ECC）检查，例如内置缓存中的RAM +> 3. 由于将脏数据从高速缓存行写回到外部存储器而触发的中止 +> +> SError被视为一种单独的异步异常类别，因为通常会针对这些情况有单独的处理程序。SError的生成是实现定义的。 +> +> **IRQ和FIQ** +> +> Arm体系结构具有两种异步异常类型，IRQ和FIQ，旨在支持外围中断的处理。它们用于信号外部事件，例如定时器触发，不代表系统错误。它们是预期的事件，与处理器指令流是异步的。IRQ和FIQ具有独立的路由控制，并且通常用于实现安全和非安全中断，如Arm通用中断控制器v3和v4指南中所讨论的。如何使用这两种异常类型是实现定义的。 + +下面是传递给inmate linux的dts中涉及到UART4的部分： + +![Screenshot_20240221_154656](20231220_linux_console_tty.assets/Screenshot_20240221_154656.png) + +然后在linux inmate cell中打开UART2和UART4的irq_chip： + +![Screenshot_20240221_155422](20231220_linux_console_tty.assets/Screenshot_20240221_155422.png) + +依然还是同样的报错Kernel panic - not syncing: Asynchronous SError Interrupt。 + +根据我在uart-inmate中的实验，推测UART4这块MMIO区域并不能读写，即使linux拿到了设备树中UART4的信息并注册了console，在真正去读写这块区域的时候就会发生内存错误。 + +## jailhouse inmate程序开发 + +写一个测试板子上uart4串口的裸机程序（.bin），并使用jailhouse启动。 + +在源码里的inmates目录仿照已有的结构创建一个新的`uart-wheatfox.c`文件。 + +![image-20240124220335289](20231220_linux_console_tty.assets/image-20240124220335289.png) + +发现一个有趣的事情，root linux启动时给串口分配的irq和手册里写的不一样…… +``` +NXP manual page 988 +irq 24 : USDHC3 +irq 26 : UART1 +irq 27 : UART2 +irq 28 : UART3 +irq 29 : UART4 +``` + +也就是说irq=38才是uart4实际的中断号 + +编写了一个最小baremetal os后，向ttymxc3的MMIO地址进行读写，但是程序立马崩溃了（读写ttymxc1的MMIO反而不会有问题，可以正常打印uart）。 + +之后我写了一个memtest，用来测试分配给这个jailhouse inmate的可读写内存区域能不能正常使用，结果出现了同样的问题。 + +![Screenshot_20240221_113843](20231220_linux_console_tty.assets/Screenshot_20240221_113843.png) + +可以看到memtest并没有成功，甚至在读这块内存时就直接没有后文了。 + +ARM64 异常定义：http://www.wowotech.net/armv8a_arch/238.html + +向ttymxc1对应的UART2打印完全没有问题： + +![Screenshot_20240221_130738](20231220_linux_console_tty.assets/Screenshot_20240221_130738.png) + +并且可以看到通过读写串口寄存器也确实把数据打印到了当前串口（这里通过ttyACM0连接了板子的UART2）。而当我去读写UART4的MMIO区域时，程序直接没有输出了： + +![Screenshot_20240221_150400](20231220_linux_console_tty.assets/Screenshot_20240221_150400.png) + +这里已经通过jailhouse提供的map_page接口添加了UART4对应的页表，但是仍然不行，推测是触发了exception，但目前jailhouse的inmate编程框架只适配了irq handle，对于其他的exception都没有处理，如果要加其他exception处理的话不太好加，需要手动修改汇编和链接文件等。 + +# 附录 + +## imx8mp的4x10pin串口连接电脑 + +打电话问forlinx技术，一种办法是焊接飞线连出来到TTL转USB的杜邦线上，另一种就是用专门的2x5 2mm间距的线，这个10pin的座叫做“牛角座”。 + +![FD7909D83503F50DF2C479CF233E5447](20231220_linux_console_tty.assets/FD7909D83503F50DF2C479CF233E5447.png) + +左上角有四个UART口，采用2x5 2mm间距，为牛角座，引脚如下： + +``` ++------+------+------+------+------+ +| NC | GND | 5V | RXD | NC | ++------+------+------+------+------+ +| NC | TXD | 5V | GND | NC | ++------+------+------+------+------+ +``` + +![7B94DDB4296F3A0016B229C6F1930890](20231220_linux_console_tty.assets/7B94DDB4296F3A0016B229C6F1930890.png) + +对于一般的TTL转USB，需要使用三个pin，即TXD, RXD和GND，上面的10pin中需要连接3pin。 + +需要注意的是，这里的pin间距只有2mm，而普通杜邦线间距是2.54mm，并且TTL转USB一般都是杜邦线母头，如下图： + +白色代表RXD（接板子TXD），绿色为TXD（接板子RXD），黑色接地。 + +![IMG_20240105_103444_edit_409390889312531](20231220_linux_console_tty.assets/IMG_20240105_103444_edit_409390889312531.jpg) + +整个流程就是： + +``` +牛角座10pin(2mm)【公头】 -> 【母头】2mm转2.54mm杜邦线【母头】 <- 【公头】杜邦线【公头】 -> 【公头】TTL转USB【USB】 +``` + +UPDATE：学长说DEBUG口可以复用两个串口，需要在板子设备树里打开第四个串口，就可以使用了。 + +``` +[ 0.152731] 30860000.serial: ttymxc0 at MMIO 0x30860000 (irq = 28, base_baud = 5000000) is a IMX +[ 0.153139] 30880000.serial: ttymxc2 at MMIO 0x30880000 (irq = 29, base_baud = 5000000) is a IMX +[ 0.153475] 30890000.serial: ttymxc1 at MMIO 0x30890000 (irq = 30, base_baud = 1500000) is a IMX +[ 1.248171] printk: console [ttymxc1] enabled +[ 1.252944] 30a60000.serial: ttymxc3 at MMIO 0x30a60000 (irq = 38, base_baud = 5000000) is a IMX +``` + diff --git a/docs/2024/20240403_loongarch_linux.md b/docs/2024/20240403_loongarch_linux.md new file mode 100644 index 0000000..38e9908 --- /dev/null +++ b/docs/2024/20240403_loongarch_linux.md @@ -0,0 +1,306 @@ +# 在hvisor(loongarch)里启动linux vm + +wheatfox 2024.3.12 + +目前选定的linux版本是6.7.9(stable 2024.3.12) +由于可能需要修改linux源码进行调试（加一些新的printk，用于测试在zone中被启动时的流程），我自己fork的linux源码位于：https://github.com/enkerewpo/linux-loongarch64 + +其中的loongson3_defconfig为我已经修改后的版本，并且对原始config进行了备份。 + +新的config我主要做了以下修改： + +1. 移除网络、USB驱动、CD-ROM驱动，以及其他暂时不用的功能，如kvm虚拟化（作为虚拟机运行的镜像没必要再开一次层虚拟化，而且本来也用不了） +2. 添加内核驱动调试输出、修改printk输出格式显示调用者信息 +3. 对irq等功能打开调试输出 + +## boot + +```c +# linux boot procedure +kernel_entry +start_kernel +``` + +## printk + +```c +pr_info/pr_error...->_printk->vprintk->vprintk_default->vprint_emit + +asmlinkage int vprintk_emit(int facility, int level, + const struct dev_printk_info *dev_info, + const char *fmt, va_list args) +{ + int printed_len; + bool in_sched = false; + + /* Suppress unimportant messages after panic happens */ + if (unlikely(suppress_printk)) + return 0; + + if (unlikely(suppress_panic_printk) && + atomic_read(&panic_cpu) != raw_smp_processor_id()) + return 0; + + if (level == LOGLEVEL_SCHED) { + level = LOGLEVEL_DEFAULT; + in_sched = true; + } + + printk_delay(level); + + printed_len = vprintk_store(facility, level, dev_info, fmt, args); + + /* If called from the scheduler, we can not call up(). */ + if (!in_sched) { + /* + * The caller may be holding system-critical or + * timing-sensitive locks. Disable preemption during + * printing of all remaining records to all consoles so that + * this context can return as soon as possible. Hopefully + * another printk() caller will take over the printing. + */ + preempt_disable(); + /* + * Try to acquire and then immediately release the console + * semaphore. The release will print out buffers. With the + * spinning variant, this context tries to take over the + * printing from another printing context. + */ + if (console_trylock_spinning()) + console_unlock(); + preempt_enable(); + } + + if (in_sched) + defer_console_output(); + else + wake_up_klogd(); + + return printed_len; +} + +-> vprintk_store +``` + +`vprintk_store`中一个比较有意思的点时它只是把需要输出的信息放在了printk ring buffer里，并不会直接通过earlycon或console进行输出，调用的是`printk_sprint`向ring buffer中保存数据。 + +由于qemu加载ELF看起来并不支持ACPI，不过看arch/loongarch里的platform init源码是默认要读一下FDT设备树的，可以先从这里开始调，如果没有设备树，linux连可用内存的范围都拿不到，之后肯定不能运行，包括从自己的allocator申请内存。 + +`early_init_fdt_scan_reserved_mem [drivers/of/fdt.c]` + +`unflatten_and_copy_device_tree` + +```c +/** + * unflatten_and_copy_device_tree - copy and create tree of device_nodes from flat blob + * + * Copies and unflattens the device-tree passed by the firmware, creating the + * tree of struct device_node. It also fills the "name" and "type" + * pointers of the nodes so the normal device-tree walking functions + * can be used. This should only be used when the FDT memory has not been + * reserved such is the case when the FDT is built-in to the kernel init + * section. If the FDT memory is reserved already then unflatten_device_tree + * should be used instead. + */ +``` + +https://elinux.org/Device_Tree_What_It_Is + +https://docs.kernel.org/admin-guide/kernel-parameters.html boot params，好像可以从这里传memory信息？ + +试着把dtb用incbin的方式塞到了vmlinux里，并且能够在fdt.c驱动中拿到这个设备树dtb。 + +但是之后调用`memblock_alloc`失败了，这里是希望申请一块给dtb的内存，然后把刚才的那个地址的传入的dtb复制到新地址（但是并没有必要啊？），这个时候linux还没拿到fdt中内存的信息，肯定没法分配的，也许和之前编译选项打开了earlyfdt有关。 + +在early fdt的时候也传入了3A5000的dtb之后，确实可以进行内存分配了，linux得到了node0的内存节点就是来自fdt中的memory信息。 + +但是printk始终没有成功输出（目前都是手动在printk_sprint时通过一个单独加的串口驱动print_char输出的，也就是说linux内核的earlycon和console都没有跑起来，对应的驱动也没有在工作）。 + +需要好好捋一遍`console_init`以及serial驱动的查找、与fdt中compatible字段的对应，以及最终拿到对应的驱动代码并启动。 + +https://www.cnblogs.com/schips/p/linux_kernel_initcall_and_module_init.html + +```c +start_kernel + -> rest_init(); + -> kernel_thread(kernel_init, NULL, CLONE_FS); + -> kernel_init() + -> kernel_init_freeable(); + -> do_basic_setup(); + -> do_initcalls(); +``` + +`trace_initcall_start` + +http://www.pedestrian.com.cn/kernel/kernel_start/kernel_start_overview.html + +console_init函数执行控制台的初始化工作，在`console_init`函数执行之前的`printk`打印信息，需要在`console_init`函数执行之后才能打印出来，在此之前`printk`的打印信息都被保存在一个缓存中（printk ring buffer）。 + +如果这里的earlycon初始化的有问题，那无论是earlycon还是正式的console就跑不起来。 + +目前我编译的vmlinux中有三个con init entry: + +```c +con_init // drivers/tty/vt/vt.c +hvc_console_init // drivers/tty/hvc/hvc_console.c +univ8250_console_init // drivers/tty/serial/8250/8250_core.c +``` + +而且是按顺序依次调用！ + +2024.3.18 earlycon跑起来了，但是只有bootconsole成功启动了，real console启动失败，这会导致之后open /dev/console时出现“无设备”报错。 + +![img](imgs/loongarch4.png) + +但是出现的问题有：vmalloc失败，tty ldisc（即line discipline终端控制）也跑不起来。先看一下vmalloc为啥分配不了。 + +## initramfs + +https://docs.kernel.org/admin-guide/initrd.html + +![img](imgs/loongarch5.png) + +目前解压和加载我自己做的rootfs.cpio.gz到设备树中声明好的initrd内存区域目前看起来没啥问题，但是到block dev驱动初始化的时候去拿这个ramdisk的时候就会出现第一次vmalloc错误： + +![img](imgs/loongarch3.png) + +这个函数调用也是相当的长： + +```c +brd_init +brd_alloc +device_add_disk +disk_scan_partitions +// 可以看到这里要去拿/dev的信息，这里和rootfs中提前准备好的dev node以及cmdline中的root=/dev/xxx都可能有关 +bdev_open_by_dev +blkdev_get_by_dev +blkdev_get_whole +bdev_disk_changed +// 然后出现了vmalloc申请内存失败，不过这个失败并不会导致驱动加载失败 +-> vmalloc failed! +``` + +加一些debug输出调一调。 + +https://www.kernel.org/doc/Documentation/blockdev/ramdisk.txt + +### unpack_to_rootfs + +https://blog.csdn.net/u012489236/article/details/103084290 讲的很详细，涉及到ramdisk和cmdline的交互 + +https://wiki.gentoo.org/wiki/Initramfs/Guide/en + +https://wiki.gentoo.org/wiki/Tmpfs/en + +## fdt + +https://elinux.org/Device_Tree_Usage + +/chosen https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt + +## unable to open initial console + +看一下内核里调用open系统调用的流程，看一下为什么/dev/console用不了 + +open(2)的flags: + +```c +/* + * FMODE_EXEC is 0x20 + * FMODE_NONOTIFY is 0x4000000 + * These cannot be used by userspace O_* until internal and external open + * flags are split. + * -Eric Paris + */ + +/* + * When introducing new O_* bits, please check its uniqueness in fcntl_init(). + */ + +#define O_ACCMODE 00000003 +#define O_RDONLY 00000000 +#define O_WRONLY 00000001 +#define O_RDWR 00000002 +#define O_CREAT 00000100 /* not fcntl */ +#define O_EXCL 00000200 /* not fcntl */ +#define O_NOCTTY 00000400 /* not fcntl */ +#define O_TRUNC 00001000 /* not fcntl */ +#define O_APPEND 00002000 +#define O_NONBLOCK 00004000 +#define O_DSYNC 00010000 /* used to be O_SYNC, see below */ +#define FASYNC 00020000 /* fcntl, for BSD compatibility */ +#define O_DIRECT 00040000 /* direct disk access hint */ +#define O_LARGEFILE 00100000 +#define O_DIRECTORY 00200000 /* must be a directory */ +#define O_NOFOLLOW 00400000 /* don't follow links */ +#define O_NOATIME 01000000 +#define O_CLOEXEC 02000000 /* set close_on_exec */ + +#define __O_SYNC 04000000 +#define O_PATH 010000000 +#define __O_TMPFILE 020000000 + +// see include/uapi/asm-generic/fcntl.h for details :) +``` + +一些ERROR NUMBER的值： + +```c +#define EPERM 1 /* Operation not permitted */ +#define ENOENT 2 /* No such file or directory */ +#define ESRCH 3 /* No such process */ +#define EINTR 4 /* Interrupted system call */ +#define EIO 5 /* I/O error */ +#define ENXIO 6 /* No such device or address */ +#define E2BIG 7 /* Argument list too long */ +#define ENOEXEC 8 /* Exec format error */ +#define EBADF 9 /* Bad file number */ +#define ECHILD 10 /* No child processes */ +#define EAGAIN 11 /* Try again */ +#define ENOMEM 12 /* Out of memory */ +#define EACCES 13 /* Permission denied */ +#define EFAULT 14 /* Bad address */ +#define ENOTBLK 15 /* Block device required */ +#define EBUSY 16 /* Device or resource busy */ +#define EEXIST 17 /* File exists */ +#define EXDEV 18 /* Cross-device link */ +#define ENODEV 19 /* No such device */ +#define ENOTDIR 20 /* Not a directory */ +#define EISDIR 21 /* Is a directory */ +#define EINVAL 22 /* Invalid argument */ +#define ENFILE 23 /* File table overflow */ +#define EMFILE 24 /* Too many open files */ +#define ENOTTY 25 /* Not a typewriter */ +#define ETXTBSY 26 /* Text file busy */ +#define EFBIG 27 /* File too large */ +#define ENOSPC 28 /* No space left on device */ +#define ESPIPE 29 /* Illegal seek */ +#define EROFS 30 /* Read-only file system */ +#define EMLINK 31 /* Too many links */ +#define EPIPE 32 /* Broken pipe */ +#define EDOM 33 /* Math argument out of domain of func */ +#define ERANGE 34 /* Math result not representable */ +``` + +测试了一下在rootfs里放了文本文件，在kernel里通过filp_open手动打开（RW）没有问题，但是打开console（设备文件，RW）时会返回一个-ENODEV（0x13，即19号错误），由于前者是和tmpfs交互（并且没有问题，包括加载ELF以及打开目录、遍历目录都没有问题），而后者实际上是要和驱动层进行交互，有一个问题就是/dev/console是在制作rootfs时手动加上的，拿这个设备如何和设备树上的serial设备绑定？这个和之前已经启动的earlycon（bootconsole）之前的关系是什么？理论上来说earlycon会在real console注册时卸载。 + +有一个需要注意的点是，rootfs里的dev设备是通过本机linux的mknode指令制作的（随后打包入cpio）： + +```bash +cd $(TMP_ROOTFS_OUTPUT)/dev && sudo mknod -m 666 null c 1 3 +cd $(TMP_ROOTFS_OUTPUT)/dev && sudo mknod -m 666 console c 5 1 +``` + +这里的console c 5 1实际上对应着char device以及主设备号、次设备的信息，所以mknode是如何制作device node节点的？这个/dev/console的二进制到底保存了什么样的信息？（能猜出来至少和一些设备相关，例如是不是character device（c）等） + +所以需要调一下do_open函数，看一下在open调用时访问/dev/console，到底是如何解析到这是一个设备node节点，并通过读取这个文件的信息拿到实际的设备进行绑定的（对/dev/console的读写在最底下实际上是对串口这个设备的寄存器的读写，写console意味者向UART打印、而读console意味者从UART读输入，不知道读输入这部分是不是会和3A5000的中断控制器有关？还是朴素的阻塞读串口的RXD FIFO？）【2024.3.24】 + +https://www.kernel.org/doc/Documentation/admin-guide/serial-console.rst 涉及到cmdline中的console是怎么和代码进行对应的 + +最后调试发现of在解析设备树的时候，对于serial驱动中的int controller parent解析有问题，在dts中去掉之后，成功进入了busybox环境，能够交互使用，启动程序： + +![loongarch1](imgs/loongarch1.jpg) + +![loongarch2](imgs/loongarch2.jpg) + +目前在qemu中不使用固件参数启动vmlinux elf基本完成，接下来就是在3A5000板子上引导启动这个ELF。 diff --git a/docs/2024/20240522_hvisor_initialization.md b/docs/2024/20240522_hvisor_initialization.md index 1a92e65..01c5d20 100644 --- a/docs/2024/20240522_hvisor_initialization.md +++ b/docs/2024/20240522_hvisor_initialization.md @@ -1,6 +1,9 @@ # Qemu的启动以及hvisor的初始化过程 +时间：2024/5/22 -*本文主要介绍了在qemu上运行hvisor和hvisor初始化过程涉及的相关知识。从qemu启动后开始跟踪整个流程，阅读完本文将对hvisor的初始化过程有一个大概的认识，具体的设计实现以及aarch64相关的内容将在其他章节给出。* +作者：徐仲锴 + +摘要：本文主要介绍了在qemu上运行hvisor和hvisor初始化过程涉及的相关知识。从qemu启动后开始跟踪整个流程，阅读完本文将对hvisor的初始化过程有一个大概的认识，具体的设计实现以及aarch64相关的内容将在其他章节给出。 ## qemu启动流程 diff --git a/docs/2024/README.md b/docs/2024/README.md index 305d697..65804fd 100644 --- a/docs/2024/README.md +++ b/docs/2024/README.md @@ -30,6 +30,16 @@ [正文](20240415_Virtio_devices_tutorial.md) +## 在hvisor(loongarch)里启动linux vm + +时间：2024年4月3日 + +作者：韩喻泷 + +摘要：介绍在龙芯3A5000硬件上通过FDT设备树启动虚拟机Linux的若干问题 + +[正文](20240403_loongarch_linux.md) + ## RISC-V APLIC 线中断总结 **时间：** 2024年3月11日 diff --git a/docs/2024/_sidebar.md b/docs/2024/_sidebar.md index 200f443..6c3b273 100644 --- a/docs/2024/_sidebar.md +++ b/docs/2024/_sidebar.md @@ -5,4 +5,4 @@ - [RISC-V APLIC 介绍](20240311_APLIC.md) - [在RuxOS上支持c++](20240229_Support_c++_on_RuxOS.md) - [在NXP OK8MP启动Jailhouse](20240223_NXP_Boot_Jailhouse_Tutorial.md) - +- [在hvisor(loongarch)里启动linux vm](20240403_loongarch_linux.md) \ No newline at end of file diff --git a/docs/2024/imgs/loongarch1.jpg b/docs/2024/imgs/loongarch1.jpg new file mode 100644 index 0000000..702377f Binary files /dev/null and b/docs/2024/imgs/loongarch1.jpg differ diff --git a/docs/2024/imgs/loongarch2.jpg b/docs/2024/imgs/loongarch2.jpg new file mode 100644 index 0000000..9b016c9 Binary files /dev/null and b/docs/2024/imgs/loongarch2.jpg differ diff --git a/docs/2024/imgs/loongarch3.png b/docs/2024/imgs/loongarch3.png new file mode 100644 index 0000000..c89c53e Binary files /dev/null and b/docs/2024/imgs/loongarch3.png differ diff --git a/docs/2024/imgs/loongarch4.png b/docs/2024/imgs/loongarch4.png new file mode 100644 index 0000000..96f8461 Binary files /dev/null and b/docs/2024/imgs/loongarch4.png differ diff --git a/docs/2024/imgs/loongarch5.png b/docs/2024/imgs/loongarch5.png new file mode 100644 index 0000000..96f8461 Binary files /dev/null and b/docs/2024/imgs/loongarch5.png differ