How do I troubleshoot traps without a line number?

[tod]

Also, everything that was said about "output" is also true of fnum. While it seems unlikely that you would overflow fnum it is possible. For example what would expect the following to do?

Code: Select all

    char fnum[20];
    double some_double = 1234567890123456789.23;
    sprintf(fnum, "%3.2f", some_double);
    cout << fnum <<'\n';

I just put this little bit of chaos in a test program and ran it. The program prints out
123456789012345678.00 and doesn't trap. But then again the test program no longer does what it's supposed to either. It corrupts memory and behaves unpredictably. Who knows maybe 35 hours from now it will trap. The inaccurate value printed is I believe a result of the limitations of doubles and has nothing to do with the overflow.

This code on the other hand:

Code: Select all

    std::ostringstream fnum;
    fnum << std::fixed << std::setprecision(8);
    double some_double = 1234567890123.12345678;
    fnum <<"fnum test:" << some_double <<'\n';
    cout << fnum.str();

prints out
fnum test:1234567890123.12353516
(again the double limitation on precision shows up) but the program both doesn't trap and it continues to operate normally. No memory was overwritten because ostringstream resizes the stream automatically. The above code does require that you #include both <sstream> and <iomanip>.

[mx270a]

Upon reading tod's link to the wiki and see how cout can replace printf, I have completely removed the printf and converted over to ostringstream. OMG is this nice. It's typesafe and I don't have to worry about buffer length. Super awesome. The prior code had been running for 4.5 days without a trap. I was going to change to sniprint as per Ridgeglider's recommendation, but ostringstream makes my code cleaner and easier to read. Now to let the system run for a a few days to see how it works.

Thanks,
Lance

[tod]

Woo hoo a convert! Now that there are two of us I think we can get tax exempt status. I also updated the wiki with some info on using istringstream instead of functions like atof() and atoi().

[mx270a]

Well, my bug is back, this time after ~12 hours of run time.

Code: Select all

-------------------Trap information-----------------------------
Exception Frame/A7 =0210F814
Trap Vector        =Access Error (2)
Format             =04
Status register SR =2000
Fault Status       =08
Faulted PC         =0206C812

-------------------Register information-------------------------
A0=00000000 A1=200009C4 A2=0210F89E A3=0210F974
A4=02114C4A A5=020690E6 A6=0210F824 A7=0210F814
D0=00000000 D1=00002000 D2=003FFFE3 D3=00000000
D4=0210F89E D5=02114C4A D6=0210F8E8 D7=0210F99C
SR=2000 PC=0206C812
-------------------RTOS information-----------------------------
The OSTCBCur current task control block = 20000A9C
This looks like a valid TCB
The current running task is: HTTP#2D
-------------------Task information-----------------------------
Task    | State    |Wait| Call Stack
Idle#3F|Ready     |    |0200C64C,0200C530,0
Main#32|Ready     |    |0200E2F2,0200276C,0200C530,0
TCPD#28|Semaphore |00BD|0200DF22,0201C6C6,0200C530,0
IP#27|Fifo      |000D|0200D3F8,02011194,0200C530,0
Enet#26|Fifo      |0024|0200D3F8,0202E4B8,0200C530,0
FTPD#30|Semaphore |0000|0200DF22,0201F754,02017948,0200C530,0
TELNET#37|Semaphore |0000|0200DF22,0201C27C,02002BAC,0200C530,0
HTTP#2D|Running   |    |0206C812,0
QUADSERIAL#2A|Semaphore |0001|0200DF22,0201F754,0200574C,0200C530,0

-------------------End of Trap Diagnostics----------------------

WinAddr2Line says 0206C812 is this again:

Code: Select all

std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&)
??:0

Code in the web.cpp file now looks like this:

Code: Select all

#include <startnet.h>
#include <stdio.h>
#include <iostream>
#include <sstream>
#include <iomanip>
#include <string>
#include <math.h>
#include <autoupdate.h>
using namespace std;

//Define External Variables
extern const char* AppName;
extern unsigned long g_UptimeSeconds;
extern bool g_USUnits;

extern int g_DirectionTTL;
extern bool g_DirectionForward;
extern string g_IOCardVersion;
extern bool g_MaxOfflineDistanceOverrideActive;

extern int g_ReceiverTTL[];
extern double g_ReceiverDistance[2][2];
extern double g_ReceiverHeading[];
extern string g_ReceiverFields[2][12];
extern double g_Distances[2][2];

extern double g_WayPointHeading[];
extern double g_WayPointDistance[];
extern bool g_WayPointStraight[];
extern int g_WayPointCount;
extern int g_ActiveWayPointSegment;
extern double g_ActiveWayPointHeading;

template<typename T>
static std::pair<bool,T> ValueFromString(std::string const& theString) {
	T value;
	std::istringstream iss (theString);
	if ((iss >> value).fail() ) return std::make_pair(false, 0);
	return std::make_pair(true, value);
}

// Functions called from a web page must be declared extern C
extern "C" {
   void DisplayFirmwareVersion(int sock, PCSTR url);
   void DisplayIOCardVersion(int sock, PCSTR url);
   void getAJAXSystemStatus(int sock, PCSTR url);
   void getAJAXPathList(int sock, PCSTR url);
   void RebootSystem();
   void ReadUserSettings();
   void WriteUserSettings();
   void DisplayUnitsDropDown(int sock, PCSTR url);
}

// Structure for storing and retieving from user Flash
#define VERIFY_VALUE (0x48666050) //A random value picked from the phone book....
struct MyOwnDataStore {
      DWORD verify_key;
      bool USUnits;
};
void ReadUserSettings() { //Read the stored data
   MyOwnDataStore *pData = (MyOwnDataStore*) GetUserParameters();
   //Verify it has the right key value. In a "Real" application, a checksum would be more robust.
   if (pData->verify_key == VERIFY_VALUE) { //Data is ok
	   if (pData->USUnits) {
		   g_USUnits = true;
		   iprintf("Loading System Settings: Using US units (feet).\r\n");
	   } else {
		   g_USUnits = false;
		   iprintf("Loading System Settings: Using metric units (meters).\r\n");
	   }
   } else {
	   iprintf("System settings could not be read from NVRAM.\r\n");
   }
}
void WriteUserSettings() {
	MyOwnDataStore mds;
	mds.USUnits = g_USUnits;
	mds.verify_key = VERIFY_VALUE;
	if (SaveUserParameters( &mds, sizeof(mds)) != 0) {
		iprintf("Parameters saved in User Parameter area.\r\n");
	} else {
		iprintf("Programming failed \r\n");
	}
}



int MyDoPost(int sock, char *url, char *pData, char *rxBuffer) {
	iprintf("----- Processing Post -----\r\n");
	iprintf("Post URL: %s\r\n", url);
	iprintf("Post Data: %s\r\n", pData);

	if (httpstricmp(url+1, "REBOOTSYSTEM")) {
		//ExtractPostData("textForm1", pData, textForm1, MAX_BUF_LEN);
		//iprintf("textForm1 set to: \"%s\"\r\n", textForm1 );
		RebootSystem();
		// Tell the web browser to issue a GET request for the next page
		RedirectResponse(sock, "advanced.htm");

	} else if (httpstricmp(url+1, "SETUNITS")) {
		char newunits[20];
		if (ExtractPostData("units", pData, newunits, 20) != -1) {
			if (!strcmp(newunits, "metric")) {
				g_USUnits = false;
				iprintf("Setting units to Metric.\r\n");
			} else if (!strcmp(newunits, "us")) {
				g_USUnits = true;
				iprintf("Setting units to Feet.\r\n");
			} else {
				iprintf("Specified units were not recognized.\r\n");
			}
			WriteUserSettings();
		}
		RedirectResponse(sock, "advanced.htm");

	} else {
		NotFoundResponse(sock, url);
		iprintf("HTTP Post Form Processor: We did not match any page\r\n");
	}
	return 1;
}
void RegisterPost() {
   SetNewPostHandler(MyDoPost);
}

void DisplayFirmwareVersion(int sock, PCSTR url) {
   writestring(sock, AppName);
}
void DisplayIOCardVersion(int sock, PCSTR url) {
	ostringstream output;
	output << g_IOCardVersion;
	writestring(sock, output.str().c_str());
}
void DisplayUnitsDropDown(int sock, PCSTR url) {
	ostringstream output;
	output << "<select name=\"units\">";
	if (g_USUnits) {
		output << "<option value=\"metric\">Metric (Meters)</option>";
		output << "<option value=\"us\" SELECTED>US (Feet)</option>";
	} else {
		output << "<option value=\"metric\" SELECTED>Metric (Meters)</option>";
		output << "<option value=\"us\">US (Feet)</option>";
	}
	output << "</select>";
	writestring(sock, output.str().c_str());
}

void getAJAXSystemStatus(int sock, PCSTR url) {
	ostringstream output;
	output << fixed; //Needed for setprecision

	unsigned long secs = g_UptimeSeconds;
	unsigned int days = secs/86400; secs -= days*86400;
	unsigned int hours = secs/3600; secs -= hours*3600;
	unsigned int mins = secs/60; secs -= mins*60;

	output << "<table border=1 cellspacing=0 cellpadding=5>";
	output << "<tr><td colspan=3>System Uptime: " << days << " Day";
	if (days != 1) output << "s";
	output << ", " << hours << " Hour";
	if (hours != 1) output << "s";
	output << ", " << mins << " Minute";
	if (mins != 1) output << "s";
	output << ", " << secs << " Second";
	if (secs != 1) output << "s";
	output << "<br><br></td></tr>";

	output << "<tr><td>Travel Direction:</td><td colspan=2";
	if (g_DirectionTTL == 0) {
		output << " bgcolor=#FF6666>I/O Card Not Available";
	} else {
		if (g_DirectionForward) {
			output << ">Forward";
		} else {
			output << ">Reverse";
		}
	}
	output << "</td></tr>";


	output << "<tr><td>-</td><td><b>Receiver 1</b></td><td><b>Receiver 2</b></td></tr>";
	output << "<tr><td><b>Receiving Data</b></td>";
	if (g_ReceiverTTL[0] == 0) {
			output << "<td bgcolor=#FF6666>No</td>";
		} else {
			output << "<td bgcolor=#00FF00>Yes</td>";
		}
	if (g_ReceiverTTL[1] == 0) {
			output << "<td bgcolor=#FF6666>No</td>";
		} else {
			output << "<td bgcolor=#00FF00>Yes</td>";
		}
	output << "</tr>";

	output << "<tr><td><b>GPS Solution Status</b></td><td>" << g_ReceiverFields[0][0] << "</td><td>" << g_ReceiverFields[1][0] << "</td></tr>";
	output << "<tr><td><b>GPS Position Type</b></td><td>" << g_ReceiverFields[0][1] << "</td><td>" << g_ReceiverFields[1][1] << "</td></tr>";
	output << "<tr><td><b>Satellites Tracked</b></td><td>" << g_ReceiverFields[0][9] << "</td><td>" << g_ReceiverFields[1][9] << "</td></tr>";
	output << "<tr><td><b>Satellites Used</b></td><td>" << g_ReceiverFields[0][10] << "</td><td>" << g_ReceiverFields[1][10] << "</td></tr>";

	output << "<tr><td><b>Vector Length</b></td><td>";
	if (g_USUnits) { //*3.28084
		std::pair<bool,double> rf02 = ValueFromString<double>(g_ReceiverFields[0][2]);
		std::pair<bool,double> rf12 = ValueFromString<double>(g_ReceiverFields[1][2]);
		if (rf02.first == true) {
			output << setprecision(2) << rf02.second * 3.28084;
		} else {
			output << "Error: Not Numeric";
		}
		output << " ft</td><td>";
		if (rf12.first == true) {
			output << setprecision(2) << rf12.second * 3.28084;
		} else {
			output << "Error: Not Numeric";
		}
		output << " ft";
	} else {
		output << g_ReceiverFields[0][2] << " m</td><td>" << g_ReceiverFields[1][2] << " m";
	}
	output << "</td></tr>";

	output << "<tr><td><b>Vector Heading</b></td><td>" << g_ReceiverFields[0][3] << "&#176;</td><td>" << g_ReceiverFields[1][3] << "&#176;</td></tr>";
	output << "<tr><td><b>Vector Pitch</b></td><td>" << g_ReceiverFields[0][4] << "&#176;</td><td>" << g_ReceiverFields[1][4] << "&#176;</td></tr>";
	output << "<tr><td><b>Heading Accuracy</b></td><td>" << g_ReceiverFields[0][6] << "&#176;</td><td>" << g_ReceiverFields[1][6] << "&#176;</td></tr>";
	output << "<tr><td><b>Pitch Accuracy</b></td><td>" << g_ReceiverFields[0][7] << "&#176;</td><td>" << g_ReceiverFields[1][7] << "&#176;</td></tr>";

	output << "<tr><td><BR><BR><b>Current Distance</b></td><td><BR><BR>";
	if (g_USUnits) {
		output << setprecision(2) << g_ReceiverDistance[0][0] * 3.28084 << " ft";
	} else {
		output << setprecision(2) << g_ReceiverDistance[0][0] << " m";
	}
	output << "</td><td><BR><BR>";
	if (g_USUnits) {
		output << setprecision(2) << g_ReceiverDistance[1][0] * 3.28084 << " ft";
	} else {
		output << setprecision(2) << g_ReceiverDistance[1][0] << " m";
	}
	output << "</td></tr>";

	output << "<tr><td><b>Offline Distance</b></td><td";
	double dbl_od = fabs(g_ReceiverDistance[0][1]);
	if (dbl_od > 0.5) {
		if (dbl_od > 1) { //Red
			output << " bgcolor=#FF6666";
		} else { //Yellow
			output << " bgcolor=#FFFF00";
		}
	}
	if (g_USUnits) {
		dbl_od *= 3.28084 * 12; //to feet, to inches
		output << ">" << setprecision(1) << dbl_od << " in";
	} else {
		output << ">" << setprecision(2) << dbl_od << " m";
	}
	output << "</td><td";

	dbl_od = fabs(g_ReceiverDistance[1][1]);
	if (dbl_od > 0.5) {
		if (dbl_od > 1) { //Red
			output << " bgcolor=#FF6666";
		} else { //Yellow
			output << " bgcolor=#FFFF00";
		}
	}
	if (g_USUnits) {
		dbl_od *= 3.28084 * 12; //to feet, to inches
		output << ">" << setprecision(1) << dbl_od << " in";
	} else {
		output << ">" << setprecision(2) << dbl_od << " m";
	}
	output << "</td></tr>";

	if (g_MaxOfflineDistanceOverrideActive) {
		output << "<tr><td colspan=3 bgcolor=#FFFF00><b>Override of Maximum Offline Distance is Enabled</b></td><tr>";
	}

	output << "</table>";
	writestring(sock, output.str().c_str());
}

void getAJAXPathList(int sock, PCSTR url) {
	ostringstream output;
	output << fixed; //Needed for setprecision

	if (g_WayPointCount == 0) {
		output << "There are no waypoints defined.<BR><BR>";
	} else {
		output << "<table border=1 cellspacing=0 cellpadding=5>";
		output << "<tr><td>Heading<BR>(in Degrees)</td><td>Distance<BR>(in ";
		if (g_USUnits) {
			output << "Feet";
		} else {
			output << "Meters";
		}
		output << ")</td><td>Path</td></tr>";

		if (g_WayPointCount > 0) {
			for (int i = 0; i < g_WayPointCount; i++) { //-1
				if (i > 0 && i == g_ActiveWayPointSegment) { //Add the row showing current location
					output << "<tr><td bgcolor=#CCCCCC><b>";
					output << setprecision(4) << g_ActiveWayPointHeading;
					output << "</b></td><td bgcolor=#CCCCCC><b>";
					if (g_USUnits) {
						output << setprecision(2) << g_ReceiverDistance[1][0] * 3.28084;
					} else {
						output << setprecision(2) << g_ReceiverDistance[1][0];
					}
					output << "</b></td><td bgcolor=#CCCCCC><b>Current Location";
					output << "</b></td></tr>";
				}

				output << "<tr><td>";
				output << setprecision(4) << g_WayPointHeading[i];
				output << "</td><td>";
				if (g_USUnits) {
					output << setprecision(2) << g_WayPointDistance[i] * 3.28084;
				} else {
					output << setprecision(2) << g_WayPointDistance[i];
				}
				output << "</td><td>";
				if (i == 0) {
					output << "-";
				} else {
					if (g_WayPointStraight[i]) {
						output << "Straight";
					} else {
						output << "Arc";
					}
				}
				output << "</td></tr>";
			}
		}
		output << "</table>";
	}
	writestring(sock, output.str().c_str());
}

I'm pretty sure it was executing either getAJAXSystemStatus() or getAJAXPathList().

[dciliske]

Out of curiosity, how large is that string in getAJAXSystemStatus? Could this be a memory fragmentation issue? Would a failed malloc cause the .c_str() call to throw an exception that isn't caught, and then trap the application? I'm out of my league if that's the case, as I've not done exception based C++ in our environment.

[mx270a]

An example of the HTML generated by getAJAXSystemStatus() is:

Code: Select all

<table border=1 cellspacing=0 cellpadding=5><tr><td colspan=3>System Uptime: 0 Days, 1 Hour, 32 Minutes, 24 Seconds<br><br></td></tr><tr><td>Travel Direction:</td><td colspan=2>Reverse</td></tr><tr><td>-</td><td><b>Receiver 1</b></td><td><b>Receiver 2</b></td></tr><tr><td><b>Receiving Data</b></td><td bgcolor=#00FF00>Yes</td><td bgcolor=#00FF00>Yes</td></tr><tr><td><b>GPS Solution Status</b></td><td>SOL_COMPUTED</td><td>SOL_COMPUTED</td></tr><tr><td><b>GPS Position Type</b></td><td>NARROW_INT</td><td>NARROW_INT</td></tr><tr><td><b>Satellites Tracked</b></td><td>16</td><td>16</td></tr><tr><td><b>Satellites Used</b></td><td>12</td><td>12</td></tr><tr><td><b>Vector Length</b></td><td>1129.95 ft</td><td>-3.28 ft</td></tr><tr><td><b>Vector Heading</b></td><td>116.4467&#176;</td><td>211.5472&#176;</td></tr><tr><td><b>Vector Pitch</b></td><td>0.0&#176;</td><td>0.0&#176;</td></tr><tr><td><b>Heading Accuracy</b></td><td>0.1111&#176;</td><td>0.1111&#176;</td></tr><tr><td><b>Pitch Accuracy</b></td><td>0.2222&#176;</td><td>0.2222&#176;</td></tr><tr><td><BR><BR><b>Current Distance</b></td><td><BR><BR>1129.95 ft</td><td><BR><BR>1130.25 ft</td></tr><tr><td><b>Offline Distance</b></td><td>0.0 in</td><td>0.0 in</td></tr></table>

That is 1232 characters long. I realize the length of some variables may make it longer, but it shouldn't be more than an additional 20 characters.

The HTML generated by getAJAXPathList() is approximately 650 characters long.

[tod]

I want to point out that . c_str() does NOT make a copy of the string, it returns a pointer to a const char[]. You are using it correctly. It should not be used to modify the underlying data and it should be assumed that further operations on the stream will invalidate it.

If you suspect std::string is leaking I would suggest adding some code to try and verify that. Per the wiki you can #include Bsp.h and add

Code: Select all

DWORD unallocated_memory = spaceleft();

Then cout the amount of free space each time (or every x times) the methods are called. If the number keeps going down you have a leak. The amount it goes down by should help as well.

[mx270a]

Memory monitoring added. Been up for an hour and the available memory is holding steady at 7,114,752.

[mx270a]

I had to give up on this issue for a while, but I'm back at it now. Recently, I've seen this trap occur anywhere from under an hour to over a day after boot up.

I don't think this is a memory leak for two reasons - the amount of run time before a trap is inconsistent, and I'm monitoring spaceleft(), which is holding steady at just over 7 MB.

WinAddr2Line is shows me this when I look up the address where the trap occurred:

Code: Select all

std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&)
??:0

I think my best method for troubleshooting at this point is to recompile stdlib with debug info so I can get a line number, and possibly do some inspection of the variables there when the trap occurs. How do I recompile stdlib so that debug info is in there and I can get a line number?

Thanks,
Lance

[pbreed]

The fact this is the string alocator is very suspicious...

Do you have ANY user written interrupt routines?


What are you other tasks doing, is there code in the stdlib string class that is not thread safe?