Pages

Thursday 2 August 2012

Results of Comparative Cross Validation Tests

As expected the NN achieved 100 % accuracy and my prediction of 20 % to 30 % accuracy for my current Naive Bayesian Classifier was more or less right - in various runs of sample sizes up to 50,000 it achieved accuracy rates of 30 % to 33 %. A screen shot of both classifiers applied to the last 200 days worth of S & P futures prices is shown below, with the Naive Bayesian in the upper pane and the NN in the lower pane.
However, despite it vastly superior performance in the tests, I don't really like the look of the NN on real data - it appears to be more erratic or noisier than the Bayesian classifier. I suspect that the NN may be overly complex, with 54 nodes in its one hidden layer. I shall try to improve the NN by reducing the number of hidden layer nodes to 25, and then seeing how that looks on real data.

2 comments:

  1. Thank you for sharing lots of brilliant information...love the posts and your attention to detail. Really excited to hear about your NN progress....and out of sample accuracy...I know this is a big commitment so have avoided using any ML so far.

    ReplyDelete
  2. @vectorman - Thanks for your kind words - it's nice to hear that my blog posts are appreciated.

    ReplyDelete