Kernel panic on shutdown

[andyolivares]

Hi:

I’m getting a Kernel panic everytime I try to shutdown my custom board. The messages I see are as follow:

debian@beaglebone:~$ sudo shutdown -h now [sudo] password for debian: [ 61.453535] reboot: Power down [ 63.940046] rtc_power_off failed, bailing out. [ 63.954628] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000 [ 63.954628] [ 63.963844] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 4.9.45-ti-r57 #1 [ 63.970929] Hardware name: Generic AM33XX (Flattened Device Tree) [ 63.977111] [] (unwind_backtrace) from [] (show_stack+0x20/0x24) [ 63.984917] [] (show_stack) from [] (dump_stack+0x80/0x94) [ 63.992203] [] (dump_stack) from [] (panic+0x100/0x28c) [ 63.999222] [] (panic) from [] (complete_and_exit+0x0/0x2c) [ 64.006588] [] (complete_and_exit) from [] (SyS_reboot+0x1c8/0x238) [ 64.014653] [] (SyS_reboot) from [] (ret_fast_syscall+0x0/0x3c) [ 64.022376] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000 [ 64.022376]

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

debian@beaglebone:~$ sudo shutdown -h now [sudo] password for debian: [   61.453535] reboot: Power down [   63.940046] rtc_power_off failed, bailing out. [   63.954628] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000 [   63.954628] [   63.963844] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 4.9.45-ti-r57 #1 [   63.970929] Hardware name: Generic AM33XX (Flattened Device Tree) [   63.977111] [] (unwind_backtrace) from [] (show_stack+0x20/0x24) [   63.984917] [] (show_stack) from [] (dump_stack+0x80/0x94) [   63.992203] [] (dump_stack) from [] (panic+0x100/0x28c) [   63.999222] [] (panic) from [] (complete_and_exit+0x0/0x2c) [   64.006588] [] (complete_and_exit) from [] (SyS_reboot+0x1c8/0x238) [   64.014653] [] (SyS_reboot) from [] (ret_fast_syscall+0x0/0x3c) [   64.022376] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000 [   64.022376]

My board boots from a microSD card (MMC0) based on latest SM-RED Linux image.
Do you have any idea what could I be missing? Maybe an entry on the device tree?

The message states something about the RTC clock but I really don’t have a clue what could be wrong. That is the only problem with my custom board I’ve noticed so far.

Find attached my device tree and my bootlog from the moment the board powers on to the moment I shut it down and I get the Kernel panic.

Thank you in advance for any help!

Attachments:

1. bootlog-1.txt
2. swms-ltb.dts_.txt

[Neeraj Dantu]

Andy,

Can you use an newer Beagle image to boot the board and see if that fixes the issue? There might be one bad process that is hanging when it is closing and that might be causing the panic. A newer kernel could have patched the bug that is causing the panic.

If you do not have a display in your system, the IOT images from Beagle’s page: https://beagleboard.org/latest-images will work better because all the GUI applications and processes are removed.

You can also build your own image. Here is a procedure to building a Beagle image: https://www.digikey.com/eewiki/display/linuxonarm/BeagleBone+Black

Best,

Neeraj

[andyolivares]

Hi Neeraj, I will try with a recent image and see if that solves the problem. I’ll get back to you with my findings. Thank you!

[andyolivares]

Hi Neeraj:

Today I tried with latest IoT image from beagleboard.org, but I’m having the same kernel panic error message:

[ 138.699523] reboot: Power down [ 141.186025] rtc_power_off failed, bailing out. [ 141.200515] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000 [ 141.200515] [ 141.209724] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 4.14.67-ti-r73 #1 [ 141.216897] Hardware name: Generic AM33XX (Flattened Device Tree) [ 141.223079] [] (unwind_backtrace) from [] (show_stack+0x20/0x24) [ 141.230882] [] (show_stack) from [] (dump_stack+0x80/0x94) [ 141.238172] [] (dump_stack) from [] (panic+0x100/0x284) [ 141.245189] [] (panic) from [] (complete_and_exit+0x0/0x2c) [ 141.252552] [] (complete_and_exit) from [] (SyS_reboot+0x1c8/0x238) [ 141.260618] [] (SyS_reboot) from [] (ret_fast_syscall+0x0/0x54) [ 141.268349] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000 [ 141.268349]

1 2 3 4 5 6 7 8 9 10 11 12 13 14

[  138.699523] reboot: Power down [  141.186025] rtc_power_off failed, bailing out. [  141.200515] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000 [  141.200515] [  141.209724] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 4.14.67-ti-r73 #1 [  141.216897] Hardware name: Generic AM33XX (Flattened Device Tree) [  141.223079] [] (unwind_backtrace) from [] (show_stack+0x20/0x24) [  141.230882] [] (show_stack) from [] (dump_stack+0x80/0x94) [  141.238172] [] (dump_stack) from [] (panic+0x100/0x284) [  141.245189] [] (panic) from [] (complete_and_exit+0x0/0x2c) [  141.252552] [] (complete_and_exit) from [] (SyS_reboot+0x1c8/0x238) [  141.260618] [] (SyS_reboot) from [] (ret_fast_syscall+0x0/0x54) [  141.268349] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000 [  141.268349]

I don’t know what could be wrong. Any ideas?

Thank you very much.

[Neeraj Dantu]

Andy,

Please check the interface between PMIC and AM335x is correct in your system. You can compare the connections on your board to those in OSD3358-SM-RED(Design files available here): https://octavosystems.com/octavo_products/osd3358-sm-red/.

Do you have the 32KHz oscillator for the RTC subsystem on your board? If so, please verify that there is a proper 32KHz clock input to the OSD335x from the oscillator. Similar issue resolved as faulty oscillator: https://e2e.ti.com/support/processors/f/791/t/612270?Linux-TPS65910-Kernel-panic-at-shutdown#pi239031349=3

Best,

Neeraj

[andyolivares]

I’ve compared both schematics and they look fine to me. This is my schematic:

https://www.dropbox.com/s/vdrj9qy9sjym0p4/pmic_i2c.png?dl=0

Also, I can confirm that I have a pretty stable 32 kHz oscillation on OSC1.

Thank you!

• This reply was modified 6 years, 2 months ago by andyolivares.

[Neeraj Dantu]

Andy,

Please see https://groups.google.com/forum/#!topic/rtc-linux/yKfIpoD5gRI for the power off procedure. The RTC module is responsible for turning off the PMIC and shutting itself off.

You could add some debug statements to omap_rtc_power_off function: https://github.com/RobertCNelson/ti-linux-kernel/blob/linux-4.14.y/drivers/rtc/rtc-omap.c#L431  to get more information about what is happening.

Please also see: https://groups.google.com/forum/#!msg/beagleboard/_vZba5Qu1FY/ezoF_EsfBAAJ. May be worth a try to set the RTC module to use internal clock.

Best,

Neeraj

[andyolivares]

Will check that out but I’m not very familiar with the whole kernel re-build process so bear with me. If you have some guidance on how to accomplish these tasks, it will be greatly appreciated. Also, to make sure I’m on the same page, you are pointing me to a specific file on kernel “4.14.y”… should I compile that specific kernel or the version present on the IoT image I’m using?

BTW, I applied Robert C Nelson patches on U-Boot to avoid EEPROM check on my board. I understand that these patches make U-Boot to assume a certain board ID and hence its configuration… maybe this configuration is not initializing RTC clock as expected?

Do you recommend on keeping this patches applied or should I program my EEPROM memory somehow with my own board ID (or compatible board ID)?

Thanks again for all the help!

[Neeraj Dantu]

Andy,

Please use the version of the kernel that you are currently working with.

As mentioned earlier, you can build a Beagle image by following the steps in https://www.digikey.com/eewiki/display/linuxonarm/BeagleBone+Black

You can also use build environments like Buildroot and Yocto to make custom images. Buildroot is the easier option. Both of them have extensive documentation and support for AM335x. Take a look at https://e-ale.org/elc2018/ for a start.

If you programmed your EEPROM after booting up, you will not need to have the patched version of u-boot. So, you can try upgrading the bootloader to see if there is an initialization issue for the RTC subsystem.

Best,

Neeraj

[andyolivares]

Hi Neeraj:

I’m revisiting this issue now that I have some more free time but still with no luck. I suspected that my problem was due to a failed RTC initialization due to my u-boot being patched to avoid the EEPROM check.

I programmed EEPROM with proper value to be recognized as a OSD3358-SM-RED reference board (as my board is pretty similar to that board except for a few missing peripherals) and I can confirm that the EEPROM had successfuly programmed those bytes. A power up after programming the EEPROM confirmed that the memory was intact. This confirms that a) the I2C bus is working just fine and b) the EEPROM memory works and retains its data.

I then booted lastest BB IoT image with success. I had to modify the uEnv.txt file to set my “dtb” file with my board’s Flattened Device Tree binary and I can see that my board is recognized as a RED board and initialized as such.

The OS seems to work fine but I’m still receiveing the same kernel panic on shutdown. If I keep the uEnv.txt unmodified (as it comes on the BB image), what DTB file is being loaded?

Can you give me some feedback on the contents of that DTB file? I’m suspecting there is something I’m not including in mine. My DTB source file is based on this one: https://github.com/octavosystems/OSD335x-Device-Tree/blob/master/OSD3358-SM-RED/osd3358-bsm-refdesign.dts but I’m not sure if that is up to date, or what’s really included on the BB image.

My actual DTB source can be seen here: https://github.com/andyolivares/dtb-rebuilder/blob/swms-ltb/src/arm/swms-ltb.dts

I want to make sure there is no configuration mistake before trying something more “advanced” like recompiling the kernel and building my own Linux image with more log entries as you previously adviced.

Thank you in advance for any help.

[andyolivares]

Just a follow up… yesterday I made my own BB linux image based on the instructions provided and it booted just fine. Although, the problem of Kernel Panic when shutting down (“sudo poweroff” or “sudo shutdown -h now”) persists. I observed that the problem does not occur when halting the system (“sudo halt”). Maybe the PMIC is not really powered off on that mode?

Other thing I noted is that I populated the 1M Ohm resistor on my RTC crystal circuit (https://www.dropbox.com/s/vzud4fzz26zgozi/rtc_clock.PNG?dl=0) but it is not being populated on the RED board. When I first saw this problem and I checked RTC oscillations with a scope, I got a 32KHz sine wave but didn’t payed much attention to the amplitude of the signal. Do you think that might be the cause? Maybe the clock is oscillating at the right frequency but not strong enough.

Will check the clock levels later today on a RED board I have and compare that with my own board to see if there’s any difference.

I’m posting all my process here just in case it will help somebody else in the future 🙂

[Neeraj Dantu]

Hey Andy,

If you don’t specify a device tree, by default the Bealgebone black device tree will be invoked (Along with some cape overlays depending on what is in uEnv.txt).

You can eliminate the device tree as the source of the error by using the Beaglebone Black device tree.

Instead of recompiling the whole kernel building the rtc_omap driver and inserting it in run-time can cut down on the debug effort here: https://github.com/RobertCNelson/ti-linux-kernel/blob/linux-4.14.y/drivers/rtc/rtc-omap.c#L431. Adding debug messages to the ‘static void omap_rtc_poweroff’ function and dumping the registers of the RTC module along with the ‘struct rtc_time tm’ to somewhere in the SD card would help understand the state the module is in. The error is probably occurring in ‘static int tm2bcd()’ (https://github.com/RobertCNelson/ti-linux-kernel/blob/linux-4.14.y/drivers/rtc/rtc-omap.c#L273) function.

Looking at the crystal, VIH and VIL for RTC_XTAL_IN are 0.65*VDDS_RTC and 0.35*VDDS_RTC. As long as the signal is oscillating across these two voltage levels, the signal is fine. However, there are specifications for the RTC oscillator circuit in “Figure 6-13. OSC1 (ZCZ Package) Crystal Circuit Schematic” (http://www.ti.com/lit/ds/symlink/am3358.pdf). You could try experimenting with values of C1 and C2 in the figure 6-13 of the datasheet.

Thanks for documenting the steps. Looking forward to the results.

Best,

Neeraj

 

[andyolivares]

Neeraj:

I really appreciate all the help.

I did more testing and to my surprise, the oscillator is not oscillating at all. I’m 100% sure I measured it in the past with a good 32kHz oscillation but now it does not oscillate. I can only see a DC voltage of about 1.5V on OSC1_IN pin.

The BB image I’m now testing (latest IoT image) is different from the one I tested when I saw the oscillator working. The problem is that I don’t remember which one I had on my SD card at that time. When I tested back on those days, the EEPROM was blank and my u-boot had some patches to ignore EEPROM checks. Now, the EEPROM has been programmed with the same values to be recognized as a RED board (as specified here: https://octavosystems.com/app_notes/osd335x-eeprom-during-boot/#_Toc382081431) and I’m testing the BB image “as is” (except that I’m putting my board’s DTB file on uEnv.txt).

So, I have a couple of questions:

1) Is the 32 kHz oscillator supposed to start oscillating at board power-on?
2) If not, who’s responsible to initialize and turn it on? u-boot? the Linux kernel somewhere?

I inspected one of RCN patches and I noticed he modified some RTC suff. In particular, when my u-boot runs, I can see the “RTC 32KCLK Source: External” (as in https://rcn-ee.com/repos/git/u-boot-patches/v2018.03/0001-am335x_evm-uEnv.txt-bootz-n-fixes.patch @ line 267). But to be honest, that doesn’t say if the clock is enabled (turned on) or not.

Also, I put debug strings on the function “rtc32k_enable()” (line 219 of the same patch) to see if the RTC was being initialized or not, but I never saw those lines during u-boot start, so I assume the function didn’t get called.
It seems that the function is only called if macro CONFIG_SPL_AM33XX_ENABLE_RTC32K_OSC is defined. I compiled u-boot with “make ARCH=arm CROSS_COMPILE=${CC} am335x_evm_defconfig” which does not define that macro. Although, the generated .config file contains a line “CONFIG_SPL_AM33XX_ENABLE_RTC32K_OSC=y”, so I’m not sure if my u-boot is being compiled with that macro turned on or not.

Anyway, I don’t understand why the RED board doesn’t need any of that and the RTC works just fine by default (that’s why I asked for the DTB file to check that there is no difference with mine on that regard). What really puzzles me is that the oscillator once worked but now it doesn’t at all.

So if you could give some answers to the two questions above, would give me more light on to where to look next. Will check the documentation you provided too, and also see if I can compile the rtc_omap driver and debug that part.

Once again, thank’s for all the help.

[Neeraj Dantu]

Andy,

To answer the question, yes the normal behavior is for the RTC subsystem to be powered on with the external oscillator.

I suspect what you are dealing with is a hardware issue with the 32KHz oscillator(It may have stopped working at some point after you probed last time?)

In order to make sure it is not a software issue, you can check/enable the external RTC crystal via software. If you look at “20.3.5.19 RTC_OSC_REG Register” in AM335x TRM(https://www.ti.com/lit/ug/spruh73p/spruh73p.pdf), bit 3 of this register selects the clock source of the RTC subsystem. You can read this register to see if the external oscillator is selected as the clock source. You can do this read in Linux using the devmem2 utility (http://manpages.ubuntu.com/manpages/bionic/man1/devmem2.1.html). Using devmem2 should work as we used it for other register level access.

If you find that the external clock source is indeed not selected, you can enable bit 3 of the register to make that selection. Note that the RTC subsystem registers are write protected and you need to write 2 KICK registers in order to get write access to the registers. Please see: 20.3.5.23 KICK0R Register and 20.3.5.24 KICK1R Register in the AM335x TRM.

Other registers of interest include:

1. RTC_CTRL_REG

2. RTC_STATUS_REG

3. RTC_SYSCONFIG

4. RTC_PMIC

Hope this helps. Please let us know what you find.

Best,

Neeraj

 

[andyolivares]

Neeraj:

I tested “devmem2” tool as suggested and it indeed says that value of register RTC_OSC_REG (address 0x44E3E054) is 0x48 which means that RTC source should be external and it should be enabled. The DC voltage level on OSC1_IN pin is about 0.9V at power up.
If I write this register with value 0x10 (disable the oscillator and apply high impedance to the output) the voltage level on OSC1_PIN jumps up to a DC level of about 1.8V.

FYI, I booted with EEPROM programed as a RED board with no special DTB file. Just latest IoT BB image as-is in order to discard DTB issues. So, to summarize, after power up, registers of interest have the following values:

RTC_OSC_REG = 0x48
RTC_CTRL_REG = 0x01
RTC_STATUS_REG = 0x02
RTC_SYSCONFIG = 0x03
RTC_PMIC = 0x10011

I also tried changing RTC_OSC_REG value to 0x40 (internal oscillator selected and enabled) and a read back confirmed the value change (as I wrote both KICK registers with unlock values), but still got the same kernel panic message. Maybe the kernel needs more than just an on-the-fly register change.

I noticed that, according to AM335x TRM, default reset value of RTC_OSC_REG is 0x10 (RTC disabled and high impedance), so there is one place where this register is being changed to 0x48… just for completeness: can you point me to the exact place where this happens? I would say that it is the rtc32k_enable() function on u-boot’s “board/ti/am335x/board.c” or is it the “omap_rtc” Linux driver.

Bottom line, it seems like I do have a hardware issue at hand. Will try removing RTC circuit components and put new ones to see if that solves the problem. I’ll keep you posted with my findings.

Thank you and happy new year BTW!

• This reply was modified 5 years, 11 months ago by andyolivares. Reason: typo
• This reply was modified 5 years, 11 months ago by andyolivares.

[andyolivares]

I’m not sure what I did different this time, but I modified RTC_OSC_REG with value 0x40 again to set internal oscillator.
I did a quick test with “sudo hwclock -f /dev/rtc0 –debug” and I got a clock tick (this command failed with external oscillator), so the RTC is working with the internal oscillator with no problems.

After that check I did a “sudo poweroff” and now the board was powered off just fine with no kernel panics. I can see my board’s power indication LED being turned off.

So it is confirmed, it is a hardware issue with my RTC oscillator circuit. Will try to fix that and see what happens.

[andyolivares]

Hi Neeraj:

I reworked my RTC circuit replacing the crystal and supporting capacitors and now it oscillates with a good 32 kHz sine wave (as can be seen on my oscilloscope). I really don’t know what could have caused the old crystal to stop working at some point.

With the RTC up and running with the external oscillator, I did a quick test again with “sudo hwclock -f /dev/rtc0 –debug” and it got a tick as expected, so running the “sudo poweroff” command turned my board off just fine with no kernel panics (finally!).

So, I think we can close this topic now and hope all my process will help someone else facing the same problem.

Thank’s again for all the help and support.

Best regards,

Andy

[Neeraj Dantu]

Andy,

Glad you could find the cause of the issue. Thanks for the documentation and closing the loop.

Best,

Neeraj

[Author]

Posts