Jose Antonio Martin H. PhD Madrid, España

"Education is what survives when what has been learned has been forgotten". B. F. Skinner

Tools:

Neural Networks and other utitlies:

• python numpy  ndlinspace,  the n-dimensional linspace function:

        Python Code of the n-dimensional linspace function nd-linspace (ndlinspace.py) (python + numpy)

• Simple and very useful Multilayer Perceptrons Neural Networks with BackPropagation training:

Python Code (bpnn.py) (pure python)

Python Code (Xbpnn.PY) (python with numeric - fast for big networks)

Java Code (bpnn.java) (pure java implementation)

Matlab simple and nice multilayer perceptron (MLP) with back-propagation training (Matlab BackPropagation) (pure Maltab/Octave implementation). This implementation is specially designed for neuro-evolution since all the weights are represented in a vector which is then automatically decoded in the evaluate function.

• Simple and very useful Kohonen's style Self Organized Maps SOM  Neural Networks:

Pure python Self Organized Map SOM SOM.py
Pure python Fuzzy Gaussian Kernel Self Organized Map OSOM OSOM.py (you must install Matplotlib only to see the example)

• An Evolutionary Neural Network (Multilayer perceptron) with Differential Evolution Training Algorithm:

Python Code (DEBPNN.py) (pure python), please download the file (DESolver.py) which is needed by the Neural Net.
This Network performs better than back propagation, also it is implemented saving and reading weights to/from file, Enjoy it.

Reinforcement Learning:

Competition Records for JAMH Team -  Reinforcement Learning Competition 

• 1st First place PolyAhtlon 2008
• 2nd Second place MountainCar 2008
• 1st First place Helicopter Hovering 2009
• 2nd Second place PolyAthlon 2009

• A Reinforcement Learning Environment in Python: (QLearning and SARSA) Version 1.0

Download the Package RLearning for python :                       ReinforcementLearning.zip   

Also, a win32 installer is provided:                                           RLearning-1.0.0.win32.exe

 

It includes as examples a Mountain Car Problem  and Cart Pole Control Problem:          

Finally I have decided to make the graphics output with vpython which I think it is the best option to produce nice and very easy to program real time graphics on python for this kind of things. After the installation, check the examples folder to see the Mountain Car demo and the Cart Pole demo.

 

 

Mountain Car	Cart Pole

 

• A Reinforcement Learning Environment in Python: (NN, kNN-TD and Exa) Version 2.0  (22 may 2010)

@Download the Package FAReinforcement for python :                       FAReinforcement   

 

It includes as examples: Acrobot,  Mountain Car.  discrete and continuous Cart Pole and a predator/protector/prey game

You should install Python+Vpython+Numpy+SciPy

note: these are High Quality/Performance Reinforcement Learning implementations! do not think they are simple just because they are public and free!.

I used this same software in the Reinforcement Learning Competitions and I have won!.

• A Reinforcement Learning Environment in Matlab: (QLearning and SARSA)

Please note that in some versions of Matlab you should delete some empty parenthesis in order avoid some errors.
This should not happen with Matlab release from version 7.
This code is a simple implementation of the SARSA Reinforcement Learning algorithm without eligibility traces, but you can easily extend it and add more features due to the simplicity and modularity of this implementation. Enjoy it!

Matlab SARSA implementation of the Mountain Car Problem:                       SARSA Mountain Car.zip

Matlab SARSA implementation of the Cart Pole Control Problem:                      SARSA CartPole.zip

Matlab SARSA implementation of the Acrobot Control Problem:                    SARSA Acrobot.zip

Matlab Multi-Agent SARSA implementation of a Three-Link Planar Robot:      SARSA Arm.zip

Matlab Multi-Agent SARSA implementation of a SCARA 3D Robot:                 SARSA-scara.zip

                           (see the paper for more information on distributed Reinforcement Learning)

 

Some pictures of the Matlab implementations:

Mountain Car:	Cart Pole:
Three-link Planar Robot:	Acrobot:

 

 

 

@ Eligibility traces:

 

Matlab SARSA(λ) (sarsa-lambda) eligibility traces implementation for the Mountain Car Problem:  SARSA lambda (eligibility traces) Mountain Car

 

 

 

 

@ The kNN-TD(λ) algorithm: the winner of the Reinforcement Learning Competition 2008, 2009

 

Matlab kNN-TD(λ) implementation for the Mountain Car Problem:  kNN-TD Mountain Car
Please note that this is a Matlab implementation, not the competition one (originally in python), and is made for academic purposes so it is not optimized for performance or software quality design. It is just an "easy to understand and easy to try" implementation. comments are welcome!.

 

 

 

 

@ The Ex<a> Continuous Actions Reinforcement Learning Algorithm. (Exa RL Algorithm)

 

Matlab Ex<a> implementation for the continuous Cart-Pole Problem: ExaCartPole.zip (updated 7/5/2010)
Please note that this is a Matlab implementation that was originally written  in python, and is made for academic purposes so it is not optimized for performance or software quality design. It is just an "easy to understand and easy to try" implementation. comments are welcome!. It can be fastened by using some knn library such as ANN.

• José Antonio Martin H. & Javier de Lope (2009), "Ex<a> : An Effective Algorithm for Continuous Actions Reinforcement Learning Problems" , In Proceedings of 35th Annual Conference of the IEEE Industrial Electronics Society (IECON 2009) . Oporto, Portugal. November 2009., pp. 2063 - 2068. IEEE.

  [DOI] [URL] [IECON 2009]

 

 

 

 

@ Partially Observable Markov Decision Processes:

A collection of experiments to prove the hypothesis of the reduction of POMDPs to MDP by means of the use of Internal Clocks.
I am sorry for not having at this time more theoretical material at hand but you can write me if you want to talk about or even better, join the rl-list at google.
  Matlab SARSA Iclock (sarsa plus internal clock) solution to Altered POMDP Mountain Car Problem:  SARSA ICLOCK Mountain Car( Altered POMDP Mountain Car Problem. The Agent cannot see the position of the car but only its speed !!!. )

Mountain Car internal clock experiment:

We will extend the list of solved problems frequently. Any comments or suggestions are welcome.

 

• Classifier System (XCS) in python: XCSPython.zip  1.0

This is a translation of Martin Butz XCSJava with some pythonic modifications.

Also, I am planning to port some parts tThis is a translation of Martin Butz XCSJava with some pythonic modifications.

Also, I am planning to port some parts to C python extension modules in order to speed up the tool using pyrex.

 

Please, report any bug.

I am working to free this implementation of any possible bug, despite that its father XCSJava seems to be bug free, this is a translation which may contain unpredictable bugs. When more tests be made I will report that this implementation is bug free. 

 

Cheers.

 

---

 

• A collection of neuro-evolution experiments in robotics  (CMA-ES and NEAT):  

             

Matlab implementation of neuro-evolution learning for robot control:	EvoNeuroControl.zip
Planar Robot	SCARA Robot

This software is part of a research paper on neuro-evolutionary methods for multi-link robots, such as the three link planar robot and the SCARA robot.

---

Comments:

---

Revised: 05/23/10.






Copyright © José Antonio Martín Hernández.

Michigan Car Accident Lawyer