neural nets + sound
the method described in the previous post doesn’t seem to work so well on other problems… i have been trying it on the “circle” problem and the “two circle” problem and it does not work very well on those. it seems the way that is working best in general now is the old standard backprop/gradient descent method that looks at every training pattern individually.
also, the new method didn’t work very well on the cascade spiral problem, either.
i’ve completed putting the quickprop algorithm into my feedforward network and i’m now running some tests on it. it seems that the method i was using in the cascade network of calculating the error on all of the training set and minimizing that does not seem to work as well as simply going through the training set and calculating the error on a single point, then adjusting the weights after each point. this is more similar to the old method i used on feedforward nets in which the weights were adjusted after each point in the training set. a {2, 4, 4, 1} net was able to match the “x” problem after 1000 epochs and get close to the solution in less than 100 using the new method, while the old method was unable to even get close within 1500 epochs. the current file is ffqp3.nb.
on another computer i’ve decided to try this new method on the old cascade network on the spiral problem and see what happens.
here are the results of the last spiral i’ll be doing with the cascade correlation algorithm. 183 hidden nodes were added and the network error was 0.290283. it trained over the entire weekend and i used much lower stopping criteria than previously, which is why it was able to add so many hidden nodes.
\[Rho] = 1.0;
\[Alpha] = 0.35;
\[Beta] = 0.35;
\[Mu] = 1.5;
\[Zeta] = 0.1;
r = 1.0;
MinError = 0.01;
MinSChange = 0.1;
MinEChange = 0.0005;
loopsize = 10;
maxweightval = 100.0;
stopafter = 1000;
candidatenodes = 8;
trained the spiral overnight again, but since no changes in the actual training algorithm were made (just changes that increased the speed) it still did poorly. however, 27 hidden nodes were added in about the same amount of time as the previous trial, in which only 16 hidden nodes were added. the network error was 0.492361.
\[Rho] = 1.0;
\[Alpha] = 0.35;
\[Beta] = 0.35;
\[Mu] = 1.5;
\[Zeta] = 0.1;
r = 1.0;
MinError = 0.01;
MinSChange = 0.1;
MinEChange = 0.0005;
loopsize = 100;
maxweightval = 20.0;
stopafter = 10000;
candidatenodes = 8;
just made some changes to the code for the cascade-quickprop network to make candidate training quite a bit quicker. previously it took about 14 seconds to go 100 times through a training set of 194 points when training the first candidate node. with the changes i’ve made it now takes about 7 seconds. the current file is cascnet24.nb.
here’s the spiral trained overnight with the most recent changes and the parameters below. the network error is 0.556669 and there are 16 hidden nodes.
\[Rho] = 0.1;
\[Alpha] = 0.35;
\[Beta] = 0.35;
\[Mu] = 1.5;
\[Zeta] = 0.1;
r = 1.0;
MinError = 0.01;
MinSChange = 0.1;
MinEChange = 0.0005;
loopsize = 100;
maxweightval = 100.;
stopafter = 10000;
candidatenodes = 10;
i will work on trying to speed up the candidate node training. there are a few spots where it seems to calculate the same things twice or more (CalcVBar, CalcEoBar, CalcSigmaO…).
currently training the spiral problem with todays changes. file is cascnet22.nb.
i caught a sign error in the network code today, in the dE/dw term, or the function called TheSTerm. fixed that and have been trying to figure out if the network will train right… it’s still sort of jumping around a lot on the output nodes, still much worse than the candidate nodes, but it generally seems to be in the right area at least (less than an error of 1, for example). here is an example of what it looks like when training the spiral… i think the “line” patterns are because i set the rho term to be very low, which results in sharp differences between the “red” and “blue” regions. therefore when the division between the regions crosses a training pattern point it will jump up or down to the next error “level”.
training of the output nodes using quickprop doesn’t seem to be working right… here is the plot of the errors after each weight adjustment:
here is the output from an overnight run on the spiral problem after the changes i described yesterday:
15 hidden nodes were added and the network’s error was 0.318231. this is an improvement in both number of nodes and error over the previous spiral test. also, it trained for a much smaller amount of time. here are the parameters used:
\[Rho] = 1.0;
\[Alpha] = 0.35;
\[Beta] = 0.35;
\[Mu] = 1.75;
\[Zeta] = 0.01;
r = 1.0;
MinError = 0.01;
MinSChange = 0.01;
MinEChange = 0.00005;
ConsecutiveSChange = 3;
ConsecutiveEChange = 6;
loopsize = 100;
maxweightval = 100.;
stopafter = 10000;
