neural nets + sound
the cascade network seems to be having the same problems it was having two summers ago, seeming to try to match all the training points but generally just getting it all wrong. i added a few status message updates to the program, so you can see how it is doing as it is training. the file cascnet6.nb is backed up on euclid. here is an image of the networks output after training with a circle pattern (i ended it before it got to the minimum error).
made some modifications to the code today between classes, didn’t have time to do any tests. mathematica 6’s “dynamic” command is very useful for displaying status messages while training (for example, how many nodes have been added, how many iterations through the training set completed, etc). also changed it so when training the outputs it will stop after the error stops changing significantly, rather then just after a set number of iterations of the training set. the file cascnet5.nb has been backed up on euclid.
finished writing the cascade training algorithm, but it doesn’t seem to be working properly. i’ll look into it tomorrow. not sure if it’s the same problem i was having two summers ago or if it is a different problem. file cascnet4.nb has been backed up on euclid.
continued working on the code for the cascade-correlation network between classes today. completed the functions to adjust candidate node weights and to return the network’s error. all that remains now is the general training algorithm. the file cascnet3.nb has been backed up on euclid. i might try to finish the code and start testing later today.
continued working on the cascade net today, but still not finished. wrote functions to return vbar and eobar, add a node to the network, output candidate node inputs, and train output nodes. still need functions to train candidate nodes, calculate network error, and the general training algorithm. file cascnet2.nb has been backed up on euclid. (switched to filezilla ftp client because coreftp was giving me problems.)
started working on the cascade-correlation network today. currently it can only give the output of a network, not train/add nodes. plan to work on the training tomorrow. as with the ff net, network structure is defined by its weights list. weights list has two elements: first is the hidden node weights list, which contains one element (which is a list of numbers) for every hidden node. the second element in the network structure list is the output node weights. one element for each output node, and each element is a list of weights. file cascnet1.nb has been backed up on euclid.
next i will begin to re-code the cascade correlation network. it is based on a paper published by fahlman and lebiere called “the cascade-correlation learning architecture”. in the summer of 2006 i tried using this network style to match the spiral pattern presented in the article, but was not able to. the file fahlman-lebiere-90.pdf has been backed up on euclid.
this spiral was trained on a {2,12,12,12,1} ff net after 5000 iterations.
did some quick tests of the ff net today between classes. also made some slight changes to the way it plots the training sets. updated file redonn3.nb has been backed up on euclid.
first image set is for a circle of “ones” with radius 5 in the center surrounded by “negative ones”. the same {2,4,4,1} structured network was used. first images is before training, second is after training. 1000 cycles of the ts of 50 points were run.
second set of images is for two different circles of “ones”. even when trained 5000 times through the network was still unable to fit the set. very little difference between the network’s output after 1000 iterations was seen at 5000 iterations. this indicates the network’s structure is unable to match this pattern.
finished writing the feedforward network, but it has not yet been thoroughly tested. the new version is much better than the old one, much more properly written, so multiple sets of weights can be stored at the same time now. network function takes two inputs: weight set and network input. training (weight-adjustment) function takes three inputs: network input, target output, and weight set.
images show an example training set for a {2,4,4,1} net. 50 training patterns, target outputs differ depending on what side of the x-axis they’re on. points show the training set, background is the network’s output. first image is the untrained network (random weights). second image is after the network was trained 100 times through the training set.
file redonn2.nb has been backed up on euclid.
worked on complete re-coding of the neural network in mathematica today. currently working on the weight adjustment function. need to keep in mind that the “position” (with input -1) weight must be adjusted as well. filename is redonn.nb, backed up on euclid.
