OSD335x C-SiP, bootloader issues

[Aleix]

Hi everyone,

After the assembly of the first boards -Thanks Octavo for the revision of the schematics – and a quick review of the asembled boards, we tried to boot the OS from an SD card, and the process is resulting rather frustrating.

We tried several methods described below, all unsuccessful. Could someone throw some light into this issue?

1. Modifying u-boot following instructions on “OSD335x EEPROM During BootRev.16/11/2018”: Altough the bootloader compiled just fine – at least that’s what it seems – There’s no output on UART0. I can see activity on I2C0, several reads to the eeprom are performed looking for the magic number.

2. After reading a while on the formus I came across a (https://octavosystems.com/forums/reply/4880/), flashed that image on an SD card and got the same result depicted in that post. I guess that verifies the connections to the SD card.

3. In an attempt to boot that very same image, we flashed the internal eeprom with a valid board following “OSD335x EEPROM During BootRev.16/11/2018”, incorrect magic number error dissapears,but the board freezes after the SPL message “U-Boot SPL 2014.04-00014-g47880f5 (Apr 22 2014 – 13:23:54)”.

4. SPL + u-boot.img loading via UART0 (http://linuxkernel51.blogspot.com/2015/08/booting-beagle-bone-black-over-uart.html). SPL seems to load properly (u-boot prints ‘C’ BUT it does not print the SPL welcome message.),when trying to load the u-boot.img transfer halts at 114kbytes.

We’re running out of ideas, is a behaviour like that expected?

Thanks!!

[Neeraj Dantu]

Hey Aleix,

1. There are several methods described in the app note that deal with the EEPROM. The ID can be hard coded as described in 4.1. U-boot can be patched to recognize a blank EEPROM as described in 4.2(Note that EEPROM needs to be blank for this method to work. A corrupted EEPROM ID will not work). You can also program the ID outside u-boot as described in section 6.

One additional thing to note is that the version of u-boot used in the app note is v2018.03. No UART output in general indicates that U-Boot was not updated. Please verify that you are updating the u-boot correctly as described in https://eewiki.net/display/linuxonarm/BeagleBone+Black(under “Setup microSD card”).

2. Yes, this verifies that you have a working SD card interface!

3. We are assuming you coded the board ID “..A335BNLTOS00” . This board ID was created a while after the 7.5 image was released. So, that image will not recognize a board with ID ending with OS00. If you code the board with beaglebone Black ID: “..A335BNLT00C0”, which was a valid ID for image version 7.5, you should be able to boot the board normally. For EEPROM IDs, please refer to: https://github.com/beagleboard/image-builder

4. We have not tried this approach, but note that booting via UART can be susceptible to data transfer issues. Please review the information above and let us know if you have any specific questions.

We are available on this forum and will be happy to help.

Best,

Neeraj

 

[Aleix]

Hi Neeraj,

I’m starting to think that the issue has little to do with the EEPROM detection, I ran a side by side capture (Beaglebone Blue vs OSD) of I2C0, and the data captured during EEPROM transfers is exactly the same, the OSD appears to halt right before starting to deal with the device at address 0x48 (0x24 7b PMIC).
Find attached the spreadsheet (maybe you guys can spot some irregularity there).

Both boards are running (swapping sd from one to other) the same SD card. I have a couple of them just to see if there are any variations. One card with version 7.5 and other with the latest image 9.5 from beagleboard.

BTW: I had already tried coding different combos of board/version based on the contents of board.h inside u-boot source… Damn it, this things are frustrating!

Attachments:

1. I2C0_BBBlue-vs-OSD-1.xlsx

[Neeraj Dantu]

Aleix,

From your latest post, it does look like it is not an EEPROM ID issue.

Given that you are able to verify that the MMC interface works and you are able to get to U-Boot and rebuild it, you can try and narrow down the exact location in U-Boot that the board is getting stuck at. This could help determine whether there is an issue with the PMIC/DDR interfaces or whether it is a software issue.

You can see board detect function is being called from here: http://git.denx.de/?p=u-boot.git;a=blob_plain;f=arch/arm/mach-omap2/am33xx/board.c;hb=HEAD. Putting some debug messages in this file to print on the UART console might help pin pointing the location of the fault.

We hope that helps. Please let us know if you have any update on this.

Best,

Neeraj

[Aleix]

Okay, that took a while and several DDings but I found part of the issue. Our board runs without the 32K RTC clock, and the SPL was freezing while executing this (note the commented lines):

#if defined(CONFIG_SPL_AM33XX_ENABLE_RTC32K_OSC)
static void rtc32k_enable(void)
{
struct davinci_rtc *rtc = (struct davinci_rtc *)RTC_BASE;

/*
* Unlock the RTC’s registers. For more details please see the
* RTC_SS section of the TRM. In order to unlock we need to
* write these specific values (keys) in this order.
*/
//writel(RTC_KICK0R_WE, &rtc->kick0r); <– COMMENTED
//writel(RTC_KICK1R_WE, &rtc->kick1r); <– COMMENTED

/* Enable the RTC 32K OSC by setting bits 3 and 6. */
//writel((1 << 3) | (1 << 6), &rtc->osc); <– COMMENTED
}

I guess it waits for a sync bit or something like that and just stays there forever. Note I commented the lines because I was unable to deselect this option from the menuconfig, looks like it’s a “standard” option. I don’t know if not initializing the RTC will have other repercussions….

Anyway, from that point on, I was able to properly load the SPL and finally to load u-boot!!! Great news after several days sitting in front of a bunch of “CCCCCCCCCCCCCCCCCC”

My joy lasted for 10-20ms, when an exeption rebooted the board. Narrowing down, it looks like some issue while loading stored variables in some nvs (I have no clue where they are actually stored).

Commenting this lines allowed u-boot to continue doing it’s thing:

‘/common/autoboot.c’

#ifdef CONFIG_BOOTCOUNT_LIMIT //CRASH HERE!!
//bootcount = bootcount_load(); <– COMMENTED
bootcount = 0; //Patch <– ADDED
bootcount++;
//bootcount_store(bootcount); <– COMMENTED
env_set_ulong(“bootcount”, bootcount);
bootlimit = env_get_ulong(“bootlimit”, 10, 0);

Now I can get up to the “starting kernel” message, where the board freezes…again…, but I guess it has something to do with not loading the kernel image/fdt ot overlapping kernel image/fdt file. I have attached a full print from the sequence. Any more ideas?

***********************************************************************
U-Boot SPL 2018.01 – MEDEA NO RTC32K – -dirty (Jun 05 2019 – 17:19:22)
Trying to boot from MMC1

U-Boot 2018.01-dirty (Jun 05 2019 – 17:16:24 +0200)

I2C: ready
DRAM: 512 MiB
No match for driver ‘omap_hsmmc’
No match for driver ‘omap_hsmmc’
Some drivers were not found
****BOARD INIT****.
Reset Source: Global external warm reset has occurred.
Reset Source: watchdog reset has occurred.
Reset Source: Power-on reset has occurred.
MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
Using default environment

****LATE INIT****.
Board: BeagleBone Black
<ethaddr> not set. Validating first E-fuse MAC
****LATE INIT DONE****.
Net: eth0: MII MODE
Could not get PHY for cpsw: addr 0
cpsw, usb_ether
****U BOOT MAIN LOOP****.
CLI init
CLI init done
preboot cmd
preboot cmd done
bootdelay_process
bootdelay process done
autoboot cmd
Press SPACE to abort autoboot in 2 seconds
board_name=[A335BNLT] …
board_rev=[00C0] …
switch to partitions #0, OK
mmc0 is current device
SD/MMC found on device 0
** Bad device 0:2 0x82000000 **
** Bad device 0:2 0x82000000 **
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1…
gpio: pin 56 (gpio 56) value is 0
gpio: pin 55 (gpio 55) value is 0
gpio: pin 54 (gpio 54) value is 0
gpio: pin 53 (gpio 53) value is 1
switch to partitions #0, OK
mmc0 is current device
gpio: pin 54 (gpio 54) value is 1
Checking for: /uEnv.txt …
Checking for: /boot.scr …
Checking for: /boot/boot.scr …
Checking for: /boot/uEnv.txt …
gpio: pin 55 (gpio 55) value is 1
2099 bytes read in 37 ms (54.7 KiB/s)
Loaded environment from /boot/uEnv.txt
Checking if uname_r is set in /boot/uEnv.txt…
gpio: pin 56 (gpio 56) value is 1
Running uname_boot …
loading /boot/vmlinuz-4.14.67-ti-r73 …
10379776 bytes read in 683 ms (14.5 MiB/s)
loading /boot/dtbs/4.14.67-ti-r73/am335x-boneblack.dtb …
60317 bytes read in 53 ms (1.1 MiB/s)
loading /boot/initrd.img-4.14.67-ti-r73 …
4530411 bytes read in 321 ms (13.5 MiB/s)
debug: [console=ttyO0,115200n8 root=/dev/mmcblk0p1 ro rootfstype=ext4 rootwait coherent_pool=1M net.ifnames=0 quiet] …
debug: [bootz 0x82000000 0x88080000:4520eb 0x88000000] …
## Flattened Device Tree blob at 88000000
Booting using the fdt blob at 0x88000000
Loading Ramdisk to 8fbad000, end 8ffff0eb … OK
Loading Device Tree to 8fb9b000, end 8fbacb9c … OK

Starting kernel …

[Neeraj Dantu]

Aleix,

Thanks for the update. This is great progress!

Since you have the RTC subsystem disabled, you will need to stop the kernel from initializing the subsystem also. This can be done through device tree. Please take a look at related thread on TI forums: https://e2e.ti.com/support/processors/f/791/t/485319?AM335x-RTC-disable-mode-SW-setting

It would be good to start to disable the RTC via device tree first to see if it works. For the device tree that you are using: am335x-boneblack.dtb(https://github.com/RobertCNelson/dtb-rebuilder/blob/4.14-ti/src/arm/am335x-boneblack.dts) RTC node is referenced in am33xx.dtsi, am335x-boneblack-common.dtsi(https://github.com/RobertCNelson/dtb-rebuilder/blob/4.14-ti/src/arm/am335x-boneblack-common.dtsi) and am335x-bone-common.dtsi(https://github.com/RobertCNelson/dtb-rebuilder/blob/4.14-ti/src/arm/am335x-bone-common.dtsi). You can remove the rtc node references in bone-common.dtsi and boneblack-common.dtsi and disable the rtc node in am33xx.dtsi. You will have to rebuild the device tree and replace the one in the file system on the SD card you are using to boot.

For further information on device tree and how to modify and rebuild a custom device tree, see: https://octavosystems.com/app_notes/osd335x-design-tutorial/osd335x-lesson-2-minimal-linux-boot/linux-device-tree/. The tool used: https://github.com/RobertCNelson/dtb-rebuilder

Another change you can make is to remove “quiet” argument from the ‘console’ environment variable defined in /boot/uEnv.txt. This will give you more verbose kernel debug messages on the UART console.

Hope this helps. Please let us know if you have an update.

Best,

Neeraj

[Aleix]

Neeraj you are the man!

After the proposed shanges I was able to boot the kernel and – finally – see some real progress, thanks!

I’ve been playing around with the system and seems stable, but I have a couple major things that do not work, the WL1831 module and the USB interface to a cellular modem.

Debugging steps taken for the WL1831:

– Verify power rails (ok)
– Verify 32.768khz clock (ok)
– Verify mii1_txd0 // BT ena(HIGH)
– Verify mii1_tx_clk //WL ena(LOW)!
– Verify kernel module is loaded (lsmod) seems consistent with the output of lsmod on a working beaglebone blue

Module: Size: USED by:
wl18xx 110592 0
wlcore 237568 1 wl18xx

I can observe few clock bursts at 400KHz on mii1_rxd1 (sdio clock to wl module) during boot (I guess it’s trying to initialize the “card”), nothing on uart3 tx/rx. I have tested the same image on our board and on the Beaglebone Blue,
and the wireless module works OK on the BBBlue. I can see both enable signals going high and activity on both uart and sdio buses.

There are some warings thrown by the dtb rebuilder at compilation, maybe it has something to do see the attached files (if you feel like testing it yourself). I have moved away all the other .dts files.

dtb rebuilder make:

DTC src/arm/am335x-boneblue.dtb
src/arm/am335x-boneblue.dtb: Warning (unit_address_format): Node /ocp/epwmss@48300000/eqep@0x48300180 unit name should not have leading “0x”
src/arm/am335x-boneblue.dtb: Warning (unit_address_format): Node /ocp/epwmss@48302000/eqep@0x48302180 unit name should not have leading “0x”
src/arm/am335x-boneblue.dtb: Warning (unit_address_format): Node /ocp/epwmss@48304000/eqep@0x48304180 unit name should not have leading “0x”
src/arm/am335x-boneblue.dtb: Warning (simple_bus_reg): Node /ocp/l4_wkup@44c00000/scm@210000/scm_conf@0/clocks missing or empty reg/ranges property
src/arm/am335x-boneblue.dtb: Warning (simple_bus_reg): Node /ocp/l4_wkup@44c00000/scm@210000/clockdomains missing or empty reg/ranges property
src/arm/am335x-boneblue.dtb: Warning (simple_bus_reg): Node /ocp/mailbox@480C8000 simple-bus unit address format error, expected “480c8000”
src/arm/am335x-boneblue.dtb: Warning (simple_bus_reg): Node /ocp/mcasp@4803C000 simple-bus unit address format error, expected “4803c000”
src/arm/am335x-boneblue.dtb: Warning (simple_bus_reg): Node /ocp/P9_19_pinmux missing or empty reg/ranges property
src/arm/am335x-boneblue.dtb: Warning (simple_bus_reg): Node /ocp/P9_20_pinmux missing or empty reg/ranges property
src/arm/am335x-boneblue.dtb: Warning (simple_bus_reg): Node /ocp/cape-universal missing or empty reg/ranges property
src/arm/am335x-boneblue.dtb: Warning (phys_property): Missing property ‘#phy-cells’ in node /ocp/usb@47400000/usb-phy@47401300 or bad phandle (referred from /ocp/usb@47400000/usb@47401000:phys[0])
src/arm/am335x-boneblue.dtb: Warning (phys_property): Missing property ‘#phy-cells’ in node /ocp/usb@47400000/usb-phy@47401b00 or bad phandle (referred from /ocp/usb@47400000/usb@47401800:phys[0])

The bluettoth interface does not work either, HCI commands only print the following. Again, no activity on uart3 bus during any of this prints/system boot.
*****
debian@beaglebone:~$ hciconfig version
hci0: Type: Primary Bus: UART
BD Address: 00:00:00:00:00:00 ACL MTU: 0:0 SCO MTU: 0:0
DOWN
RX bytes:0 acl:0 sco:0 events:0 errors:0
TX bytes:16 acl:0 sco:0 commands:4 errors:0
******
And a bunch of errors HCI-related the console throws during startup

[ 36.331871] NET: Registered protocol family 31
[ 36.387606] Bluetooth: HCI device and connection manager initialized
[ 36.467742] Bluetooth: HCI socket layer initialized
[ 36.547568] Bluetooth: L2CAP socket layer initialized
[ 36.611823] Bluetooth: SCO socket layer initialized
[ 37.144555] Bluetooth: HCI UART driver ver 2.3
[ 37.209994] Bluetooth: HCI UART protocol H4 registered
[ 37.346076] Bluetooth: HCI UART protocol LL registered
[ 37.613843] Bluetooth: HCI UART protocol ATH3K registered
[ 37.859176] Bluetooth: HCI UART protocol Three-wire (H5) registered
[ 38.037270] Bluetooth: HCI UART protocol QCA registered
[ 38.610466] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 38.633955] Bluetooth: BNEP filters: protocol multicast
[ 38.650961] Bluetooth: BNEP socket layer initialized
[ 39.519578] Bluetooth: hci0 command 0x1001 tx timeout
[ 47.583428] Bluetooth: hci0: Reading TI version information failed (-110)
[ 47.590323] Bluetooth: hci0: download firmware failed, retrying…
[ 49.727424] Bluetooth: hci0 command 0x1001 tx timeout

Regarding the USB interface to the cellular modem, I guess it’s some sort of configuration of the OS, it only lists the root hub and it did create a couple usb-eth adapters via software. Any insights on how to gain control over the USB ports?

Thanks!

Attachments:

1. dtb-rebuilder-src-arm.zip

[Aleix]

Let me attach also the full console prints from reset to linux login.

Attachments:

1. boot-console-out.txt

[Neeraj Dantu]

Aleix,

Good to hear on getting the board to boot!

Based on your post and the schematics/layout we have, it doesn’t look like there is anything wrong. Some inputs on debug:

1. Check for any shorts on the SDIO and UART/BT data lines. It looks like the driver comes up and the kernel modules are inserted. But, WL_EN goes low and that could be because the processor could not load proper firmware for the module.

2. Check the part orientation on MOD1(WL1831). Pin 1 should be on the lower right corner of the module.

3. Since you are using the WL1831 as opposed to WL1835 on the Beaglebone Blue, you might have to change the compatibility string in the device tree(https://github.com/torvalds/linux/blob/master/Documentation/devicetree/bindings/net/wireless/ti%2Cwlcore.txt). There were no firmware loading differences that we could notice in /lib/firmware/ti-connectivity for different modules, but the HCI command timeouts are suspicious.

4. I was able to build the device tree without issue on a Beaglebone Blue. Note that you will need dtb-rebuilder version 4.9 vs master(4.14): https://github.com/RobertCNelson/dtb-rebuilder/tree/4.9-ti

 

Coming to the cellular modem, relevant boot message: “[    3.487453] usb 1-1: device descriptor read/64, error -71”. This indicates that the module did not respond to USB enumeration. Please verify voltages on the module and make sure that it is not being held under reset. In particular, +3V3_CELL needs to be ~3.8V. As you have access to SARA_RESET and SARA_POWER_ON, you could try and reset the module manually to see if it comes up.

Hope this helps, as always, let us know if you have an update.

Best,

Neeraj

 

[Aleix]

Hi Neeraj

I’m stuck XD could you please provide the dtb rebuilder folder you used to do the changes please??

Thanks!

[Neeraj Dantu]

Aleix,

Please find it attached here
[Update] Looks like the file is too big. Google Drive link instead: https://drive.google.com/open?id=1zVDnXMCFixzQWZXQgELNrt6Ysjj0UfVp

Best,

Neeraj

• This reply was modified 4 years, 11 months ago by Neeraj Dantu.

[Aleix]

Hi Neeraj!

Thanks for the files, I cross checked your changes against mines and they were the same, the issue lied on one silly pull up resitor on the wlan_irq line. Cross checking my schematics against the BBBlue schematics, the pull-up reistor on that particular line is marked as DNP, removing it made the module work 🙂

Again, there’s another issue, I cannot boot from the internal emmc, I executed the emmc flasher script (by uncommenting the variable in /boot/uEnv.txt), and the emmc loads everything correctly, however it does not seem to execute the bootloader upon power-up. If I execute the bootloader from the Sd card and then remove the SD card, the system boots correctly from the emmc.

I tried manually loading the MLO and u-boot.img to mmcblk1 to the same offsets as in the SD but it makes no difference. Any tips there?

To future readers encountering similar issues, I’ve compiled a “guide” outlining the steps to boot the SIP and to disable the RTC nodes. I hope it helps. Neeraj, if it’s not too much to ask, could you please review it?

Thanks!

[Aleix]

I cannot attach the file (1.4MB vs 512KB)… I’m sharing the google drive link instead!

https://docs.google.com/document/d/1mLe8aKckKvg9QwjWJkdTg8gPltMKCRK__lkcNBFa6k0/edit

[Neeraj Dantu]

Great and thanks for sharing the guide with the community! This is much appreciated 🙂

Coming to the eMMC booting issue, the first thing to check is the boot configuration resisters. LCD_DATA[4:0] are latched during boot sequence to determine which interfaces need to be probed for the second stage bootloader(SPL) and in what order. See table 26-7 in TRM(https://www.ti.com/lit/ug/spruh73p/spruh73p.pdf) for boot sequence on each setting.

In your design, if OSD_BOOT_SD# is being pulled low, the boot sequence would not include MMC1 which is the interface the eMMC is on. This might result in the RBL(ROM Bootloader) not checking the eMMC for SPL. So, please make sure that T2 is not ON pulling the LCD_DATA2 signal LOW.

Hope that helps. Keep us updated!

Best,

Neeraj

[Aleix]

Hi Neeraj,

I had to leave the project aside for a short while, apologies for getting back to you so late.

After carefully checking the OSD_BOOT_SD# upon power up, the line does not appear to be pulled low, however there’s no way I can manage the device to boot from the emmc.

I confirm that the line does make electrical connection to the device itself (pulling high/low does force the device to boot from the SD or from the eMMC), however when the eMMC is selected as boot device i get zip on the UART.

Do you have of any possible (even remote) idea of what might be going on there? We are using the generic eMMC flasher script on the standard beagleboard released images, is there any tweak we need to apply to that script?

Is it possible that the device not booting is in any way related to the changes made in u-boot and device tree?

Thanks,

[Neeraj Dantu]

Aleix,

The flasher script should copy all the contents of the SD card to the eMMC. If you can verify activity on the eMMC interface by probing the CMD and CLK lines of the MMC1 interface, that would confirm that the board is trying to boot from the eMMC.

Another thing to verify is whether the eMMC is flashed. You can boot from the SD card and mount the eMMC to browse and verify the right root file system.

Based on your description of nothing on the UART0(normally, if the processor does not find a boot source it will print “CCCC” on UART0 console), it looks like the unpatched(without your updates for disabling the RTC) version of u-boot is getting executed. I know you mentioned that you overwrote the u-boot multiple times manually. One thing you can try is to erase the blocks of eMMC that contain the u-boot as described in “Erase partition table/labels on microSD card:”(needs to be done to the eMMC instead of the SD card) in https://eewiki.net/display/linuxonarm/BeagleBone+Black and then flash the eMMC using the updated SD card you have.

Hope this helps. Please let us know if you have any updates.

Best,

Neeraj

[Author]

Posts