Spontaneous reboot while flying and hanging shutdown issues

[Reacher10]

My clock just stopped so I went to change the volume, I could change the volume setting but there was no volume change. I turned the radio off and back on, the volume changed as it should have, pressed long up and the date and time text read WAR: 00:00:00. It was still correct in the menu. I reset the volume to max, turned it on and off and the long up showed date/time was correct and running.

[cmanley]

My FPV vacation is over and I'm currently testing my radio with the same old firmware, without a TX module plugged in, to see if I can reproduce a reboot.
I did find a dubious connection in one of my LiFe battery wires, probably caused by the strain on the stiff wires when (dis)connecting the JST-XH 3 pin connector from the 9X. If a reboot happens again with the different battery I'm using now, I'll post an update, and then take it step by step from there. My coprocessor version is 0004 btw.

[Reacher10]

Haven't had another reboot yet in 2.5 hours at the faster reading rate!

The coprocessor TinyApp.bin version 02 is somewhere on the repository, just need to find which revision it belongs to (might be 116).

Reacher10: You can't easily erase the co-processor. You would need to program it directly (not using SAM-BA) with the bootloader. The ARM will still try to read the co-processor though.

Mike.

Well I'll just leave it as is for now. Remember I never had any reboots before so if I have to erase the co precessor to stop the reboots I'll do it later down the line.

Clock stopped again, went thru the same proceedure, same results. It actually reads "00-War- 00:00:00:00".

This time I left the volume on the lower setting and turned it off and on and everything read correctly.

[bertrand35]

openTx doesn't run the mixer in the same thread (than the menus, the coproc, the SD ...), perhaps it helps. Unless this is an uncleared interrupt problem of course!

[Reacher10]

My FPV vacation is over and I'm currently testing my radio with the same old firmware, without a TX module plugged in, to see if I can reproduce a reboot.
I did find a dubious connection in one of my LiFe battery wires, probably caused by the strain on the stiff wires when (dis)connecting the JST-XH 3 pin connector from the 9X. If a reboot happens again with the different battery I'm using now, I'll post an update, and then take it step by step from there. My coprocessor version is 0004 btw.

You need to go to radio setup menu page 5 to see the co processor version. It should be 010?.

[cmanley]

That's where I went and mine is "Co Proc 0004".

[Reacher10]

Boom...just had a reboot

[MikeB]

I seem to be able to freeze the RTC reading with my test code quite quickly. I think there must be a problem with the driver code for this, possibly with two failure modes, one that freezes (most often) and one that causes the reboot (occasionally).
I wonder if there are any reports on openTx where the RTC freezes and you can't change the volume?

Testing continues.

Mike.

[Reacher10]

Mike, can you give me a quck how to on erasing the co processor and getting it back to where it was, it is verB board from the first 25 sent out so I don't think the co processor had anything on...not even the boot loader. Then I'll let it run and see if I get any reboots.

[Rob Thomson]

This help

http://openrcforums.com/forum/viewtopic ... 517#p25544

[MikeB]

Quick answer, try reading from here:
http://openrcforums.com/forum/viewtopic ... 270#p24925

I'll need to look up the exact connections needed if you can't see them from that.

Ahh, I knew there was some better info somewhere.

Mike.

[ShowMaster]

My main tx is a ersky9x ver c board with RTC/battery backup. It has the upgraded LCD display and switching regulator mod installed by SkyNorth. To date with 15-20 min flights of various planes I've never seem a reboot that I know of. I did load r162 a week ago but haven't flown with it yet. From all I'm reading should at least use the most current FW now.
I have voice on but not a lot of announcements used right now.
I use a 3 c life battery, Frsky and haptic mods. No BT module but I do have the rotary encoder in place of one tx pot.
I've flown this setup as a beta test for the ver 3 board release without issues.
I intend to keep flying it but I'll maybe use my er9x m64 tx for higher cost planes for now?
In fact I just orders a SMD voice upgrade board for my er9x m64 tx. I had the parts and mp3 board to bound it but for $45 drop in I'm going to forgo the fun of jamming all the discrete parts on a broad and in my 9x.
PNP this time.
Back to the sky boards,
I also have a ver B board that has the heatsink mods and runs on a 2c lipo. It too is a beta board but has been a solid performer since the sd card holder replacement. Of course I'm not flying for hours at a time but I've not had any reboots or hangs on powering off when I do fly it. I fly many 15-20 minute flights in a row, turning the tx off between flights.
I'm following the troubleshooting going on here for answers. Hopefully there will be a breakthrough is what's going on soon.
SM



Sent from my iPod touch using Tapatalk.

[MikeB]

SM: I would go for r165 now, that has the 'best' reboot time (0.5 secs) if there are any problems.

I getting fairly certain the problem is somewhere in the co-processor driver and/or the co-processor itself. Today, with my frequent access to the RTC I have had several reboots and RTC freezes. I feel sure that if I sort a fix, a few hours in this mode, of trouble free running, will confirm the fix is OK.

I'm using 05 co-processor code.

When the bus froze, I found SCL high and SDA low. I reset the co-processor only and it cleared the freeze, RTC resumed working.

Mike.

[Reacher10]

Can I just chisel the co processor off the main board...wood chisel is sharp.

[jhsa]

Do you think is safe?

[MikeB]

Can I just chisel the co processor off the main board...wood chisel is sharp.

You could try linking pin 8 to pin 14 of the 14-pin header beside the co-processor. That will hold it permanently reset, nearly as good as removing it!.

Mike.

[Reacher10]

I could make a jumper from Pin 8 to 14 and thats it? If I remove the jumper then the ATTINY 167 would just be running again as before?

I mean it's worth trying to see if the reboots stop.

EDIT: OK...I hooked up the jumper and the Co proc on page 5 reads FF00, I can't set the time or date and the long press up shows 00-War- 00:00:00:00. If I access page 5 then long press up it shows 01-Jan- 00:00:00:00.

I remove the jumper and everything is back to normal.

I'll just leave the jumper on and see what happens.

[MikeB]

Great, useful test. I've got a fix for the silly "War" display. I'm detecting certain I2C failures, getting some debug to help see what's wrong.

Mike.

[kaos]

wow, a lot of posts since last night.
Mine coproc is 004. it never had any problem before but r165 made it stop. so there is some connection there too.
good to know we are onto something. This thing has been bugging for quite a while.

Mike: should I reload r165 to clear the 'watchdog' or there is a way to clear it? or just keep it running like it is now with the watchdog on the page?

also, since this started to happen on mine around dec 20th, I wonder if Atteny 004 was put out around that time. if so, then it explains the timing it started to happen.

[RCHH]

Well ... I have joined the re-boot club! I updated the copro to 05 and made no other changes (firmware r162)... Had my tea and Boink! It rebooted during tea - less than half an hour switched on doing nothing. I reverted the copro to 02 (you send me that back along Mike and it was still on my PC) and the system is sat here quite happy. I will change back to copro 05 in a few hours and see if it re-boots again. Could be narrowing it down now ...

[Rob Thomson]

That is no doubt why I have never had the issue. I have not updated my copro!

[Reacher10]

Kaos, just turn your radio off and on and the "watchdog" msg will go away and then read "power on" untill it reboots again.

[MikeB]

Co-processor App revisions:
02 - first version released 28-Sep-2012 (r116)
03 - 3-Oct-2012 (r118)
04 - 17-Oct-2012 (r122)
05 - 21-Nov-2012 (r123)
But who knows if/when anyone actually updated their co-processor.

I have various possible fixes running, with debug to log if they catch anything. Still hitting the co-processor every 20mS to speed up testing.
I am seeing some logged things, so some things are being detected.
I'm trying to get a fix that does not involve a co-processor update. If the RTC stops updating then OK, as long as there are NO reboots!

Mike.

[kaos]

Kaos, just turn your radio off and on and the "watchdog" msg will go away and then read "power on" untill it reboots again.

Thx, found that out just a while ago.

Ah, copro 004 was made in Oct. I know I did not update it right away. I must have updated it some time in Dec and spotted the reboot on Dec 20th.

[MikeB]

Just found a small bug in the co-processor app. It is supposed to timeout any I2C transactions, but it doesn't always work. It can allow the I2C bus to hang if certain things go wrong. This should, however, only cause the RTC and volume to stop operating, not cause a reboot!

Mike.

[kaos]

got another 'watchdog' with number '2'. this time I did not know, was doing something else in another room, just out of curiosity checked the 'boot reason' page to find out. without the r165 I would never know there had been a reboot.

[slowjett]

I've had a similar issue. The Turnigy 9x just rebooted on me in flight today. I got it last year from nitro planes. No idea on the revision, but I'm using STOCK firmware. Not a looses connection of bad switch.

[MikeB]

With the I2C interface to the co-processor, there are three possible sources of interrupt. At the end of the interrupt, it was supposed to disable all three, however I had an bug in the code and one of them was not disabled (something unused was disabled instead). Should that interrupt source be set, when not expected, it would cause the interrupt to run continuously, resulting in a watchdog reboot.
Since I've changed the code to disable it properly, I haven't had any reboots, although the RTC has stopped.
Just re-charging the Tx battery, but I may not get much testing run time today.

The same, incorrect, code is on openTx.

Code: Select all

  TWI0->TWI_IDR = TWI_IDR_TXCOMP | TWI_IDR_TXBUFE | TWI_PTCR_TXTDIS ;

should be

Code: Select all

	TWI0->TWI_IDR = TWI_IDR_TXCOMP | TWI_IDR_TXBUFE | TWI_IDR_RXBUFF ;
	TWI0->TWI_PTCR = TWI_PTCR_TXTDIS | TWI_PTCR_RXTDIS ;	// Stop transfers

although I have some other changes, including a transaction timeout, being added.

Mike.

[MikeB]

Over an hour of trouble free running, no reboots, clock still running. This probably equates to 12.5 hours of normal running with my rapid co-processor accessing.
4 unexpected events have been logged that might have caused either a reboot, or the clock to have stopped.

This is with an updated co-processor application as well as with ARM code changes.

Mike.

[Reacher10]

I've had a similar issue. The Turnigy 9x just rebooted on me in flight today. I got it last year from nitro planes. No idea on the revision, but I'm using STOCK firmware. Not a looses connection of bad switch.

Hi Slowjett,

The issue we are having would not be related to what you are experiencing. We are all using a custom main board called the Sky9 and running the custom firmware ERSKY.

I know I read a post where someone with a stock main board was having shut downs but I believe it was the switch...I would search for that post or thread.

-----------------------------------------------------------------------

I've have had my radio running since yesterday when I disabled the Attiny 167 but lets see...it was 3 1/2 days between reboots the last time. I'll tell ya what, with the module removed it runs a long time.