changes

made some changes to the code as suggested by andy today. first changed error calculations to use the square instead of absolute value. also changed quickprop to add the dE/dw term. current file is cascnet19.nb.

after training over the weekend

i let the network train on the spiral problem over the weekend but it wasn’t able to solve it in that amount of time. 26 hidden nodes were added and the final network error was 0.67639. here is the output:

the weights to produce this output are:

{{{-3.6692, 39.7561, -100}, {6.70387, -4.08576, 93.2557, 33.5011}, {-0.0872343, -22.0564, 55.7126, 100, 86.0025}, {-4.40567, 4.43443, -7.44347, -0.635202, -100, 2.95467}, {11.9379, 5.50393, -40.4731, 19.9817, 36.7759, -100, 0.534887}, {36.8225, 3.91947, -84.9846, 15.8074, 8.76565, 1.04606, -8.42797, -70.6273}, {75.1905, -41.0201, 100, -100, 20.2759, 100, -100, 35.2546, -51.4397}, {-0.210347, -1.24339, 30.0143, 46.9925, -95.071, -10.1429, -80.7174, 72.2197, 99.0635, -3.20148}, {-18.6181, 1.45621, -100, 100, 5.46103, -2.80784, 15.9126, 24.5726, -9.14596, 98.8905, -24.7495}, {-3.79546, 0.800163, -0.406996, -0.754752, 0.816613, 0.619208, -0.662707, 14.3088, -11.8432, -10.0736, -100, 0.701137}, -10.6882, 20.5601, 16.5673, -10.2839, 27.3703, 44.162, 15.4716, 99.2738, 2.48383, 2.35074, 62.9091, -99.9994, -50.317}, {0.0694653, -0.698207, -1.80505, -38.5157, 39.2548, -100, -100, -4.38413, 4.06213, -6.35638, -12.9872, -7.91862, -27.6211, 0.582994}, {56.0666, 7.17084, -58.0134, -28.8259, -68.2923, -63.2819, -100, -100, 3.44287, 100, -45.4276, -90.6208, -95.9426, -8.07638, -38.3196}, {-20.6714, 1.16067, -47.1209, 87.5899, 64.5487, 1.84962, 2.62063, 0.128569, 7.2581, 28.3763, 7.06693, 12.241, -4.02059, 73.8481, -10.4676, -0.924688}, {-3.17381, -0.53425, 90.8483, 47.0461, 46.6027, 8.055, 57.1758, -0.326802, -3.41636, -100, 92.6323, -85.795, -13.615, 35.2334, -92.7562, 67.8561, 73.6682}, {-2.06778, -2.41021, -2.79989, -3.61444, 13.4916, 2.04127, -11.5353, 2.3377, -2.04778, -5.95447, -0.381711, -41.7298, -3.2932, -14.5488, 2.92587, 31.2919, -82.7588, 4.88987}, {-14.2459, 23.1424, -95.5182, 36.4629, -14.6671, -20.9518, -16.4582, 16.574, -99.5474, 33.7706, 81.3634, -20.464, 9.59216, -78.3929, -3.03677, -4.11515, -14.0388, 88.482, 1.42332}, {4.32858, 0.602887, -23.201, -11.8316, -3.28485, -14.1148, -16.981, -35.9979, 61.7868, 84.6517, -14.472, -10.2875, -44.8713, -26.9056, -24.4907, -16.317, 100, -77.9609, -50.4387, -7.48718}, {-5.35465, -4.15026, -24.5604, 40.3641, -15.7932, -34.7019, -28.8796, -0.108024, 95.1992, 7.31162, 22.9023, 1.90613, -0.730991, 14.0786, 44.5835, 23.0956, -12.9014, 35.5448, 96.3168, -53.7108, 27.6284}, {24.6289, 21.7757, 2.01817, -26.1209, 56.8568, -15.3682, -14.1018, -12.5612, -18.3992, -94.0147, 14.326, 1.9896, -31.7341, -36.1899, -21.1023, 59.1751, -44.2933, 15.7361, -21.7727, -23.1874, -97.8475, -45.9864}, {23.1776, -11.5271, 99.1145, 17.6687, 54.7435, 68.6865, 85.1402, 3.93395, -17.9383, 10.4244, -23.1146, 63.2006, -4.2996, -6.4977, -97.8101, 37.3798, -44.9508, 25.6367, 19.4417, -2.25762, -17.1214, 100, -7.68719}, {-4.6861, 7.56583, -1.80056, 100, 20.6504, 34.985, 12.6636, -4.56275, 4.91744, -94.4271, -100, -51.6643, -12.348, -97.9032, 72.2741, 94.5929, -99.9999, -98.6668, -27.9413, -14.7231, -13.1223, 91.6636, 62.3009, -83.4958}, {5.18852, -17.2896, 100, 99.4683, -11.4426, 2.13289, -31.0968, 11.175, -59.8095, -0.0317132, -8.05348, 37.4432, 14.5113, 89.5923, -33.3137, -3.92199, -53.8604, -69.977, 39.5426, 11.546, 43.8239, -6.08289, 97.2248, 100, -58.3974}, {22.3436, 46.6734, 4.56659, 61.2962, 100, 2.27883, 27.5656, -0.715437, 1.17552, 2.49543, -1.05661, 1.99826, 54.6173, -100, -5.39798, -2.91017, 65.4241, -100, -100, 97.896, -12.9241, -3.06287, 2.34916, -0.108613, 100, 10.2167}, {3.75661, 14.8774, 3.88487, -78.2937, -12.0452, 1.0796, 2.77375, 34.7131, 59.0667, -0.237225, 3.09677, -0.80861, -0.415416, -0.935071, 1.32451, 1.34882, 9.97789, -90.3362, -42.2774, -0.176538, -0.162741, -55.6783, -98.0815, -95.043, -95.8813, -5.85288, -26.0335}, {1.71243, -0.485851, -5.28846, -2.92789, -3.18875, -2.16651, -3.42943, 0.0983419, -0.682936, -1.24371, 0.0339251, -4.66488, 7.09638, 16.6936, 2.35922, 3.1288, -3.58367, -1.10626, 3.19621, -2.33932, -1.38673, -2.26523, -5.85856, -2.75936, -3.95574, -21.3231, -2.84657, 1.22823}}, {{-1.72007, -0.295631, 10.5973, -7.41381, 8.60357, 8.03801, 4.6801, 10.7625, -2.65959, 11.2252, 52.5714, 67.4191, 26.3695, 21.8138, 32.8369, 28.5954, 7.94995, 24.9247, 39.2901, 21.932, -26.4095, -58.9141, 55.1241, -40.9168, 72.2804, -23.6544, 14.817, -18.1394, 4.84015}}}

fixing output weights

fixing output weights does not seem to be working very well. almost no progress at all seems to be made even when there are good weights trained for the candidate node correlation.

success?

i implemented the limit on weights (set to +/-100 right now) and was able to solve when i’ll call the “x” problem with 5 hidden nodes and a network error of 0.00182. this file is cascnet17.nb.

i also made some changes (not yet tested) to another file (cascnet18.nb), which fixes all output weights after they have been trained once. in other words, new hidden nodes only have their connections to the output nodes trained right after they are connect. i’ll be testing this next. both files have been backed up on euclid.

the error

mathematica just gave the same error when i was trying to calculate the correlation term “S”. Where S is the function i’ve defined, ts and ta are training set and training answers (256 patterns), the following code will lag the computer for a few minutes and produce an error:

S[ts, ta, {{}, {{7.331188109551801*^6, 4.982383970152668*^7, -80.25937828406873}}}, {1.32189574322, 2.13980434532, 3.3298290523}]

however, this code evaluates instantly:

S[ts, ta, {{}, {{7.331188109551801, 4.982383970152668, -80.25937828406873}}}, {1.32189574322, 2.13980434532, 3.3298290523}]

the problems seems to come in when large numbers (on the order of 10^6, 10^7) are used in the calculations. placing a limit on how high the weights can get seems like it might be a good thing to try out.

at home and at school

at home: that one candidate finally finished training but didn’t do anything at all to help the network’s error. i stopped network training because it didn’t seem to be going anywhere.

at school: the good news is that the changes made worked and the computer didn’t reset overnight. the bad new is that mathematica had an error. “No more memory available. Mathematica kernel has shut down. Try quitting other applications and then retry.” since the kernel shut down, the weights at the time of the error can not be restored.

more changes to the cascade net

so far today i’ve made a few more changes to the quickprop-cascade network. i changed it so it outputs the candidate node’s output (like those pictures shown in the fahlman paper on cascade), i made it so while training both candidate and output nodes it will keep track of what weights gave the lowest error or highest correlation (because quickprop tends to make the error/correlation jump around a lot, and sometimes it will hit a good spot but go back down or up immediately), then use that set of weights no matter what they were at then end of training (most of the time they will just be the last ones), and finally i adjusted the candidate training algorithm so that three different candidate nodes are trained, and it uses the one with highest correlation.

so far all of my tests have shown that this network right now is very unpredictable. sometimes the training will slow down very much and training has to be halted (this usually happens after adding a candidate to the network), and sometimes when training the candidate nodes the correlation (“S”) will jump down to zero and stay there forever. i still do not know why either of these things are happening. i am not tracking the weight changes for the output nodes after every adjustment, to try to find anything that could cause such huge slowdowns. i suspect it might come from the fact that some of the weights are adjusted to very large numbers and mathematica might have troubles dealing with them. i think i remember reading something somewhere about putting a cap on the values of the weights, maybe i should look into that more.

slow

the candidate node i posted about on monday is still training, but getting closer to the limit for moving on. hopefully it will be done by tomorrow, maybe even when i wake up.

first day back at school

so far today i’ve worked on adding a quickprop-like algorithm for the training of the candidate nodes in the cascade network and changed the way that the network determines when to stop training a candidate or when to add a new candidate. previously it would stop if 3 consecutive changes in either error or the correlation term “S” were small enough. now it stops if the difference between the averages of the errors or “S” terms of two consecutive sets over 100 runs through the training set is small enough. there seems to be a problem now in training the candidate nodes, though, because the “S” term always becomes zero after a few runs through. i’m going to now look for what could be causing the problem. most current file is cascnet15.nb.

31 december 2007

messing around with the quickprop-cascade net some more… it has been training a single candidate node weightset for about 24 hours now… maybe i should try to implement quickprop on the candidate training as well? here is the output after 4 nodes added, currently working on adding the fifth node.