speeding up candidate training

just made some changes to the code for the cascade-quickprop network to make candidate training quite a bit quicker. previously it took about 14 seconds to go 100 times through a training set of 194 points when training the first candidate node. with the changes i’ve made it now takes about 7 seconds. the current file is cascnet24.nb.

overnight spiral…

here’s the spiral trained overnight with the most recent changes and the parameters below. the network error is 0.556669 and there are 16 hidden nodes.

\[Rho] = 0.1;
\[Alpha] = 0.35;
\[Beta] = 0.35;
\[Mu] = 1.5;
\[Zeta] = 0.1;
r = 1.0;
MinError = 0.01;
MinSChange = 0.1;
MinEChange = 0.0005;
loopsize = 100;
maxweightval = 100.;
stopafter = 10000;
candidatenodes = 10;

for tomorrow

i will work on trying to speed up the candidate node training. there are a few spots where it seems to calculate the same things twice or more (CalcVBar, CalcEoBar, CalcSigmaO…).

currently training the spiral problem with todays changes. file is cascnet22.nb.

today

i caught a sign error in the network code today, in the dE/dw term, or the function called TheSTerm. fixed that and have been trying to figure out if the network will train right… it’s still sort of jumping around a lot on the output nodes, still much worse than the candidate nodes, but it generally seems to be in the right area at least (less than an error of 1, for example). here is an example of what it looks like when training the spiral… i think the “line” patterns are because i set the rho term to be very low, which results in sharp differences between the “red” and “blue” regions. therefore when the division between the regions crosses a training pattern point it will jump up or down to the next error “level”.

training output nodes

training of the output nodes using quickprop doesn’t seem to be working right… here is the plot of the errors after each weight adjustment:

spiral overnight after changes made

here is the output from an overnight run on the spiral problem after the changes i described yesterday:

15 hidden nodes were added and the network’s error was 0.318231. this is an improvement in both number of nodes and error over the previous spiral test. also, it trained for a much smaller amount of time. here are the parameters used:

\[Rho] = 1.0;
\[Alpha] = 0.35;
\[Beta] = 0.35;
\[Mu] = 1.75;
\[Zeta] = 0.01;
r = 1.0;
MinError = 0.01;
MinSChange = 0.01;
MinEChange = 0.00005;
ConsecutiveSChange = 3;
ConsecutiveEChange = 6;
loopsize = 100;
maxweightval = 100.;
stopafter = 10000;

changes

made some changes to the code as suggested by andy today. first changed error calculations to use the square instead of absolute value. also changed quickprop to add the dE/dw term. current file is cascnet19.nb.

after training over the weekend

i let the network train on the spiral problem over the weekend but it wasn’t able to solve it in that amount of time. 26 hidden nodes were added and the final network error was 0.67639. here is the output:

the weights to produce this output are:

{{{-3.6692, 39.7561, -100}, {6.70387, -4.08576, 93.2557, 33.5011}, {-0.0872343, -22.0564, 55.7126, 100, 86.0025}, {-4.40567, 4.43443, -7.44347, -0.635202, -100, 2.95467}, {11.9379, 5.50393, -40.4731, 19.9817, 36.7759, -100, 0.534887}, {36.8225, 3.91947, -84.9846, 15.8074, 8.76565, 1.04606, -8.42797, -70.6273}, {75.1905, -41.0201, 100, -100, 20.2759, 100, -100, 35.2546, -51.4397}, {-0.210347, -1.24339, 30.0143, 46.9925, -95.071, -10.1429, -80.7174, 72.2197, 99.0635, -3.20148}, {-18.6181, 1.45621, -100, 100, 5.46103, -2.80784, 15.9126, 24.5726, -9.14596, 98.8905, -24.7495}, {-3.79546, 0.800163, -0.406996, -0.754752, 0.816613, 0.619208, -0.662707, 14.3088, -11.8432, -10.0736, -100, 0.701137}, -10.6882, 20.5601, 16.5673, -10.2839, 27.3703, 44.162, 15.4716, 99.2738, 2.48383, 2.35074, 62.9091, -99.9994, -50.317}, {0.0694653, -0.698207, -1.80505, -38.5157, 39.2548, -100, -100, -4.38413, 4.06213, -6.35638, -12.9872, -7.91862, -27.6211, 0.582994}, {56.0666, 7.17084, -58.0134, -28.8259, -68.2923, -63.2819, -100, -100, 3.44287, 100, -45.4276, -90.6208, -95.9426, -8.07638, -38.3196}, {-20.6714, 1.16067, -47.1209, 87.5899, 64.5487, 1.84962, 2.62063, 0.128569, 7.2581, 28.3763, 7.06693, 12.241, -4.02059, 73.8481, -10.4676, -0.924688}, {-3.17381, -0.53425, 90.8483, 47.0461, 46.6027, 8.055, 57.1758, -0.326802, -3.41636, -100, 92.6323, -85.795, -13.615, 35.2334, -92.7562, 67.8561, 73.6682}, {-2.06778, -2.41021, -2.79989, -3.61444, 13.4916, 2.04127, -11.5353, 2.3377, -2.04778, -5.95447, -0.381711, -41.7298, -3.2932, -14.5488, 2.92587, 31.2919, -82.7588, 4.88987}, {-14.2459, 23.1424, -95.5182, 36.4629, -14.6671, -20.9518, -16.4582, 16.574, -99.5474, 33.7706, 81.3634, -20.464, 9.59216, -78.3929, -3.03677, -4.11515, -14.0388, 88.482, 1.42332}, {4.32858, 0.602887, -23.201, -11.8316, -3.28485, -14.1148, -16.981, -35.9979, 61.7868, 84.6517, -14.472, -10.2875, -44.8713, -26.9056, -24.4907, -16.317, 100, -77.9609, -50.4387, -7.48718}, {-5.35465, -4.15026, -24.5604, 40.3641, -15.7932, -34.7019, -28.8796, -0.108024, 95.1992, 7.31162, 22.9023, 1.90613, -0.730991, 14.0786, 44.5835, 23.0956, -12.9014, 35.5448, 96.3168, -53.7108, 27.6284}, {24.6289, 21.7757, 2.01817, -26.1209, 56.8568, -15.3682, -14.1018, -12.5612, -18.3992, -94.0147, 14.326, 1.9896, -31.7341, -36.1899, -21.1023, 59.1751, -44.2933, 15.7361, -21.7727, -23.1874, -97.8475, -45.9864}, {23.1776, -11.5271, 99.1145, 17.6687, 54.7435, 68.6865, 85.1402, 3.93395, -17.9383, 10.4244, -23.1146, 63.2006, -4.2996, -6.4977, -97.8101, 37.3798, -44.9508, 25.6367, 19.4417, -2.25762, -17.1214, 100, -7.68719}, {-4.6861, 7.56583, -1.80056, 100, 20.6504, 34.985, 12.6636, -4.56275, 4.91744, -94.4271, -100, -51.6643, -12.348, -97.9032, 72.2741, 94.5929, -99.9999, -98.6668, -27.9413, -14.7231, -13.1223, 91.6636, 62.3009, -83.4958}, {5.18852, -17.2896, 100, 99.4683, -11.4426, 2.13289, -31.0968, 11.175, -59.8095, -0.0317132, -8.05348, 37.4432, 14.5113, 89.5923, -33.3137, -3.92199, -53.8604, -69.977, 39.5426, 11.546, 43.8239, -6.08289, 97.2248, 100, -58.3974}, {22.3436, 46.6734, 4.56659, 61.2962, 100, 2.27883, 27.5656, -0.715437, 1.17552, 2.49543, -1.05661, 1.99826, 54.6173, -100, -5.39798, -2.91017, 65.4241, -100, -100, 97.896, -12.9241, -3.06287, 2.34916, -0.108613, 100, 10.2167}, {3.75661, 14.8774, 3.88487, -78.2937, -12.0452, 1.0796, 2.77375, 34.7131, 59.0667, -0.237225, 3.09677, -0.80861, -0.415416, -0.935071, 1.32451, 1.34882, 9.97789, -90.3362, -42.2774, -0.176538, -0.162741, -55.6783, -98.0815, -95.043, -95.8813, -5.85288, -26.0335}, {1.71243, -0.485851, -5.28846, -2.92789, -3.18875, -2.16651, -3.42943, 0.0983419, -0.682936, -1.24371, 0.0339251, -4.66488, 7.09638, 16.6936, 2.35922, 3.1288, -3.58367, -1.10626, 3.19621, -2.33932, -1.38673, -2.26523, -5.85856, -2.75936, -3.95574, -21.3231, -2.84657, 1.22823}}, {{-1.72007, -0.295631, 10.5973, -7.41381, 8.60357, 8.03801, 4.6801, 10.7625, -2.65959, 11.2252, 52.5714, 67.4191, 26.3695, 21.8138, 32.8369, 28.5954, 7.94995, 24.9247, 39.2901, 21.932, -26.4095, -58.9141, 55.1241, -40.9168, 72.2804, -23.6544, 14.817, -18.1394, 4.84015}}}

fixing output weights

fixing output weights does not seem to be working very well. almost no progress at all seems to be made even when there are good weights trained for the candidate node correlation.

success?

i implemented the limit on weights (set to +/-100 right now) and was able to solve when i’ll call the “x” problem with 5 hidden nodes and a network error of 0.00182. this file is cascnet17.nb.

i also made some changes (not yet tested) to another file (cascnet18.nb), which fixes all output weights after they have been trained once. in other words, new hidden nodes only have their connections to the output nodes trained right after they are connect. i’ll be testing this next. both files have been backed up on euclid.