Kernel panic on shutdown

Hi Neeraj, I will try with a recent image and see if that solves the problem. I’ll get back to you with my findings. Thank you!

Hi Neeraj:

Today I tried with latest IoT image from beagleboard.org, but I’m having the same kernel panic error message:

I don’t know what could be wrong. Any ideas?

Thank you very much.

I’ve compared both schematics and they look fine to me. This is my schematic:

https://www.dropbox.com/s/vdrj9qy9sjym0p4/pmic_i2c.png?dl=0

Also, I can confirm that I have a pretty stable 32 kHz oscillation on OSC1.

Thank you!

• This reply was modified 6 years, 1 month ago by andyolivares.

Will check that out but I’m not very familiar with the whole kernel re-build process so bear with me. If you have some guidance on how to accomplish these tasks, it will be greatly appreciated. Also, to make sure I’m on the same page, you are pointing me to a specific file on kernel “4.14.y”… should I compile that specific kernel or the version present on the IoT image I’m using?

BTW, I applied Robert C Nelson patches on U-Boot to avoid EEPROM check on my board. I understand that these patches make U-Boot to assume a certain board ID and hence its configuration… maybe this configuration is not initializing RTC clock as expected?

Do you recommend on keeping this patches applied or should I program my EEPROM memory somehow with my own board ID (or compatible board ID)?

Thanks again for all the help!

Hi Neeraj:

I’m revisiting this issue now that I have some more free time but still with no luck. I suspected that my problem was due to a failed RTC initialization due to my u-boot being patched to avoid the EEPROM check.

I programmed EEPROM with proper value to be recognized as a OSD3358-SM-RED reference board (as my board is pretty similar to that board except for a few missing peripherals) and I can confirm that the EEPROM had successfuly programmed those bytes. A power up after programming the EEPROM confirmed that the memory was intact. This confirms that a) the I2C bus is working just fine and b) the EEPROM memory works and retains its data.

I then booted lastest BB IoT image with success. I had to modify the uEnv.txt file to set my “dtb” file with my board’s Flattened Device Tree binary and I can see that my board is recognized as a RED board and initialized as such.

The OS seems to work fine but I’m still receiveing the same kernel panic on shutdown. If I keep the uEnv.txt unmodified (as it comes on the BB image), what DTB file is being loaded?

Can you give me some feedback on the contents of that DTB file? I’m suspecting there is something I’m not including in mine. My DTB source file is based on this one: https://github.com/octavosystems/OSD335x-Device-Tree/blob/master/OSD3358-SM-RED/osd3358-bsm-refdesign.dts but I’m not sure if that is up to date, or what’s really included on the BB image.

My actual DTB source can be seen here: https://github.com/andyolivares/dtb-rebuilder/blob/swms-ltb/src/arm/swms-ltb.dts

I want to make sure there is no configuration mistake before trying something more “advanced” like recompiling the kernel and building my own Linux image with more log entries as you previously adviced.

Thank you in advance for any help.

Just a follow up… yesterday I made my own BB linux image based on the instructions provided and it booted just fine. Although, the problem of Kernel Panic when shutting down (“sudo poweroff” or “sudo shutdown -h now”) persists. I observed that the problem does not occur when halting the system (“sudo halt”). Maybe the PMIC is not really powered off on that mode?

Other thing I noted is that I populated the 1M Ohm resistor on my RTC crystal circuit (https://www.dropbox.com/s/vzud4fzz26zgozi/rtc_clock.PNG?dl=0) but it is not being populated on the RED board. When I first saw this problem and I checked RTC oscillations with a scope, I got a 32KHz sine wave but didn’t payed much attention to the amplitude of the signal. Do you think that might be the cause? Maybe the clock is oscillating at the right frequency but not strong enough.

Will check the clock levels later today on a RED board I have and compare that with my own board to see if there’s any difference.

I’m posting all my process here just in case it will help somebody else in the future 🙂

Neeraj:

I really appreciate all the help.

I did more testing and to my surprise, the oscillator is not oscillating at all. I’m 100% sure I measured it in the past with a good 32kHz oscillation but now it does not oscillate. I can only see a DC voltage of about 1.5V on OSC1_IN pin.

The BB image I’m now testing (latest IoT image) is different from the one I tested when I saw the oscillator working. The problem is that I don’t remember which one I had on my SD card at that time. When I tested back on those days, the EEPROM was blank and my u-boot had some patches to ignore EEPROM checks. Now, the EEPROM has been programmed with the same values to be recognized as a RED board (as specified here: https://octavosystems.com/app_notes/osd335x-eeprom-during-boot/#_Toc382081431) and I’m testing the BB image “as is” (except that I’m putting my board’s DTB file on uEnv.txt).

So, I have a couple of questions:

1) Is the 32 kHz oscillator supposed to start oscillating at board power-on?
2) If not, who’s responsible to initialize and turn it on? u-boot? the Linux kernel somewhere?

I inspected one of RCN patches and I noticed he modified some RTC suff. In particular, when my u-boot runs, I can see the “RTC 32KCLK Source: External” (as in https://rcn-ee.com/repos/git/u-boot-patches/v2018.03/0001-am335x_evm-uEnv.txt-bootz-n-fixes.patch @ line 267). But to be honest, that doesn’t say if the clock is enabled (turned on) or not.

Also, I put debug strings on the function “rtc32k_enable()” (line 219 of the same patch) to see if the RTC was being initialized or not, but I never saw those lines during u-boot start, so I assume the function didn’t get called.
It seems that the function is only called if macro CONFIG_SPL_AM33XX_ENABLE_RTC32K_OSC is defined. I compiled u-boot with “make ARCH=arm CROSS_COMPILE=${CC} am335x_evm_defconfig” which does not define that macro. Although, the generated .config file contains a line “CONFIG_SPL_AM33XX_ENABLE_RTC32K_OSC=y”, so I’m not sure if my u-boot is being compiled with that macro turned on or not.

Anyway, I don’t understand why the RED board doesn’t need any of that and the RTC works just fine by default (that’s why I asked for the DTB file to check that there is no difference with mine on that regard). What really puzzles me is that the oscillator once worked but now it doesn’t at all.

So if you could give some answers to the two questions above, would give me more light on to where to look next. Will check the documentation you provided too, and also see if I can compile the rtc_omap driver and debug that part.

Once again, thank’s for all the help.

Neeraj:

I tested “devmem2” tool as suggested and it indeed says that value of register RTC_OSC_REG (address 0x44E3E054) is 0x48 which means that RTC source should be external and it should be enabled. The DC voltage level on OSC1_IN pin is about 0.9V at power up.
If I write this register with value 0x10 (disable the oscillator and apply high impedance to the output) the voltage level on OSC1_PIN jumps up to a DC level of about 1.8V.

FYI, I booted with EEPROM programed as a RED board with no special DTB file. Just latest IoT BB image as-is in order to discard DTB issues. So, to summarize, after power up, registers of interest have the following values:

RTC_OSC_REG = 0x48
RTC_CTRL_REG = 0x01
RTC_STATUS_REG = 0x02
RTC_SYSCONFIG = 0x03
RTC_PMIC = 0x10011

I also tried changing RTC_OSC_REG value to 0x40 (internal oscillator selected and enabled) and a read back confirmed the value change (as I wrote both KICK registers with unlock values), but still got the same kernel panic message. Maybe the kernel needs more than just an on-the-fly register change.

I noticed that, according to AM335x TRM, default reset value of RTC_OSC_REG is 0x10 (RTC disabled and high impedance), so there is one place where this register is being changed to 0x48… just for completeness: can you point me to the exact place where this happens? I would say that it is the rtc32k_enable() function on u-boot’s “board/ti/am335x/board.c” or is it the “omap_rtc” Linux driver.

Bottom line, it seems like I do have a hardware issue at hand. Will try removing RTC circuit components and put new ones to see if that solves the problem. I’ll keep you posted with my findings.

Thank you and happy new year BTW!

• This reply was modified 5 years, 10 months ago by andyolivares. Reason: typo
• This reply was modified 5 years, 10 months ago by andyolivares.

I’m not sure what I did different this time, but I modified RTC_OSC_REG with value 0x40 again to set internal oscillator.
I did a quick test with “sudo hwclock -f /dev/rtc0 –debug” and I got a clock tick (this command failed with external oscillator), so the RTC is working with the internal oscillator with no problems.

After that check I did a “sudo poweroff” and now the board was powered off just fine with no kernel panics. I can see my board’s power indication LED being turned off.

So it is confirmed, it is a hardware issue with my RTC oscillator circuit. Will try to fix that and see what happens.

Hi Neeraj:

I reworked my RTC circuit replacing the crystal and supporting capacitors and now it oscillates with a good 32 kHz sine wave (as can be seen on my oscilloscope). I really don’t know what could have caused the old crystal to stop working at some point.

With the RTC up and running with the external oscillator, I did a quick test again with “sudo hwclock -f /dev/rtc0 –debug” and it got a tick as expected, so running the “sudo poweroff” command turned my board off just fine with no kernel panics (finally!).

So, I think we can close this topic now and hope all my process will help someone else facing the same problem.

Thank’s again for all the help and support.

Best regards,

Andy