edu.iastate.jrelm.rl
Class SimpleStatelessPolicy<I,A extends Action>

java.lang.Object
  extended by edu.iastate.jrelm.rl.AbstractStatelessPolicy<I,A>
      extended by edu.iastate.jrelm.rl.SimpleStatelessPolicy<I,A>
Type Parameters:
I - - the type used to identify Actions.
A - - the type of Action used.
All Implemented Interfaces:
Policy<I,A,java.lang.Object,State>, StatelessPolicy<I,A>
Direct Known Subclasses:
REPolicy

public class SimpleStatelessPolicy<I,A extends Action>
extends AbstractStatelessPolicy<I,A>

A simple implementation of the StatelessPolicy interface. This is essentially a discrete probability distribution governing the choice of Action from a given ActionDomain, irrespective of the state of the world. To be used with a ReinforcementLearner implementing a stateless algorithm.

Author:
Charles Gieseler

Field Summary
protected  java.util.ArrayList<I> actionIDList
           
protected  ActionDomain<I,A> domain
           
protected  SimpleEventGenerator eventGenerator
           
protected  A lastAction
           
protected  double[] probDistFunction
          Here a probability distribution function (pdf) is an array of probability values.
protected  cern.jet.random.engine.RandomEngine randomEngine
           
 
Constructor Summary
SimpleStatelessPolicy(ActionDomain<I,A> actionDomain)
          Construct a SimpleStatelessPolicy using a given ActionDomain.
SimpleStatelessPolicy(ActionDomain<I,A> actionDomain, double[] initProbs)
          A new MersenneTwister seeded with the current time ((int)System.currentTimeMillis()) is created as the RandomEngine for this policy.
SimpleStatelessPolicy(ActionDomain<I,A> actionDomain, double[] initProbs, int randSeed)
           
SimpleStatelessPolicy(ActionDomain<I,A> actionDomain, double[] initProbs, cern.jet.random.engine.RandomEngine randomGen)
          Note: RandomGenerator does not reveal the seed value being used.
SimpleStatelessPolicy(ActionDomain<I,A> actionDomain, int randSeed)
          Construct a SimplePolicy using the given ActionDomain and psuedo-random generator seed.
SimpleStatelessPolicy(ActionDomain<I,A> actionDomain, cern.jet.random.engine.RandomEngine randomGen)
          Construct a SimpleStatelessPolicy using a given ActionDomain and RandomEngine.
 
Method Summary
 A generateAction()
          Choose an Action according to the current probability distribution function.
 ActionDomain<I,A> getActionDomain()
          Get the set of actions this policy uses.
 double[] getDistribution()
          Retrieve the probability distribution used in selecting actions from the action domain.
 A getLastAction()
          Get the last action chosen by this policy.
 int getNumActions()
          Retrieve the number of possible actions in the DiscreteFiniteDomain for this policy.
 double getProbability(I actionID)
          Gets the current probability of choosing an action.
 int getRandomSeed()
           
protected  void init()
           
 void reset()
          Reset this policy.
 void setDistribution(double[] distrib)
          Set the probability distribution used in selecting actions from the action domain.
 void setProbability(I actionID, double newValue)
          Updates the probability of choosing the indicated Action
 void setRandomEngine(cern.jet.random.engine.RandomEngine engine)
          Sets the RandomEngine to be used by this policy.
 void setRandomSeed(int seed)
          Resets the RandomEngine, initializing it with the given seed.
 
Methods inherited from class edu.iastate.jrelm.rl.AbstractStatelessPolicy
generateAction, getProbability, getStateDomain, setProbability
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

probDistFunction

protected double[] probDistFunction
Here a probability distribution function (pdf) is an array of probability values. When used in conjuction with the eventGenerator, a value indicates the likelihood that its index will be selected. Each Action has an ID and each Action ID has an index in the list of IDs kept by the ActionDomain. The corresponding index in this probability distribution function contains a probability value for that ID. So we have a mapping from probabilities to Action IDs and from Action IDs to Actions. This allows the evenGenerator to use the pdf to select Actions from the ActionDomain according to the specified probability distribution. The probability values are modified by a ReinforcementLearner according to the implemented learning algorithm.


randomEngine

protected cern.jet.random.engine.RandomEngine randomEngine

eventGenerator

protected SimpleEventGenerator eventGenerator

domain

protected ActionDomain<I,A extends Action> domain

actionIDList

protected java.util.ArrayList<I> actionIDList

lastAction

protected A extends Action lastAction
Constructor Detail

SimpleStatelessPolicy

public SimpleStatelessPolicy(ActionDomain<I,A> actionDomain)
Construct a SimpleStatelessPolicy using a given ActionDomain. Note, this policy requires a finite domain of discrete actions. A new MersenneTwister seeded with the current time ((int)System.currentTimeMillis()) is created as the RandomEngine for this policy. Note: If creating multiple SimpleStatelessPolicies and MersenneTwister is the desired RandomEngine, it will be more efficient to create a single MersenneTwister and pass it to each new policy as it is constructed.

Parameters:
actionList - - the collection of possible actions this policy learns over
See Also:
MersenneTwister

SimpleStatelessPolicy

public SimpleStatelessPolicy(ActionDomain<I,A> actionDomain,
                             int randSeed)
Construct a SimplePolicy using the given ActionDomain and psuedo-random generator seed. A new MersenneTwister seeded with the given seed value is created as the RandomEngine for this policy. Note: If creating multiple SimpleStatelessPolicies and MersenneTwister is the desired RandomEngine, it will be more efficient to create a single MersenneTwister and pass it to each new policy as it is constructed.

Parameters:
aDomain - - the collection of possible Actions
sDomain - - the collection of possible States
randSeed - - seed value for the random generator used in this policy
See Also:
MersenneTwister

SimpleStatelessPolicy

public SimpleStatelessPolicy(ActionDomain<I,A> actionDomain,
                             cern.jet.random.engine.RandomEngine randomGen)
Construct a SimpleStatelessPolicy using a given ActionDomain and RandomEngine. Note, this policy requires a finite domain of discrete actions. The policy will use the given RandomEngine in selecting new choices. Note: RandomGenerator does not reveal the seed value being used. As such, SimpleStatelessPolicy has no way of knowing what seed is being used when a pre-constructed generator is supplied. SimpleStatlessPolicy will set the seed -1 and report this value when getRandomSeed() is called.

Parameters:
actionList - - the collection of possible actions this policy learns over
randomGen - - the RandomEngine this policy will use

SimpleStatelessPolicy

public SimpleStatelessPolicy(ActionDomain<I,A> actionDomain,
                             double[] initProbs)
A new MersenneTwister seeded with the current time ((int)System.currentTimeMillis()) is created as the RandomEngine for this policy. Note: If creating multiple SimplePolicies and MersenneTwister is the desired RandomEngine, it will be more efficient to create a single MersenneTwister and pass it to each new policy as it is constructed.

See Also:
MersenneTwister

SimpleStatelessPolicy

public SimpleStatelessPolicy(ActionDomain<I,A> actionDomain,
                             double[] initProbs,
                             int randSeed)
                      throws java.lang.IllegalArgumentException
Throws:
java.lang.IllegalArgumentException

SimpleStatelessPolicy

public SimpleStatelessPolicy(ActionDomain<I,A> actionDomain,
                             double[] initProbs,
                             cern.jet.random.engine.RandomEngine randomGen)
                      throws java.lang.IllegalArgumentException
Note: RandomGenerator does not reveal the seed value being used. As such, SimpleStatelessPolicy has no way of knowing what seed is being used when a pre-constructed generator is supplied. SimpleStatlessPolicy will set the seed -1 and report this value when getRandomSeed() is called.

Parameters:
actionDomain -
initProbs -
randomGen -
Throws:
java.lang.IllegalArgumentException
Method Detail

init

protected void init()

generateAction

public A generateAction()
Choose an Action according to the current probability distribution function.

Returns:
a new Action

reset

public void reset()
Reset this policy. Reverts to a uniform probability distribution over the domain of actions. This only modifies the probability distribution. It does not reset the RandomEngine. WARNING: This will effectivlely erase all learned Action probabilities in this policy.


getDistribution

public double[] getDistribution()
Retrieve the probability distribution used in selecting actions from the action domain.

Returns:
the current probability distribution over the domain of actions.

setDistribution

public void setDistribution(double[] distrib)
                     throws java.lang.IllegalArgumentException
Set the probability distribution used in selecting actions from the action domain. The distribution is given as an array of doubles. Note: there should be a value for each Action in this policy's ActionDomain. Each value is associated with the Action

Parameters:
distrib - - the new collection of action choice probabilities
Throws:
java.lang.IllegalArgumentException

getActionDomain

public ActionDomain<I,A> getActionDomain()
Get the set of actions this policy uses.

Returns:
the associated ActionDomain
See Also:
edu.iastate.jrelm.rl.StatelessPolicy#getDomain()

getNumActions

public int getNumActions()
Retrieve the number of possible actions in the DiscreteFiniteDomain for this policy.

Returns:
size of the ActionDomain

getLastAction

public A getLastAction()
Get the last action chosen by this policy.

See Also:
StatelessPolicy.getLastAction()

getProbability

public double getProbability(I actionID)
Gets the current probability of choosing an action. Parameter actionIndex indicates which action in the policy's domain to lookup. If the given actionIndex is not within the bounds of the domain, Not A Number is returned (Double.NaN).

Parameters:
actionIndex - - int indicator of which action to look up
Returns:
the probability of the specified Action. Double.NaN if action index is out of bounds for the domain of actions.
See Also:
Policy#getProbability(SI, AI)

setProbability

public void setProbability(I actionID,
                           double newValue)
Updates the probability of choosing the indicated Action

Parameters:
actionID - - the index of the desired Action in this policy's ActionDomain.
newValue - - new choice probability value to associate with this action.
See Also:
Policy#setProbability(SI, AI, double)

setRandomEngine

public void setRandomEngine(cern.jet.random.engine.RandomEngine engine)
Sets the RandomEngine to be used by this policy.

Parameters:
engine -

getRandomSeed

public int getRandomSeed()

setRandomSeed

public void setRandomSeed(int seed)
Resets the RandomEngine, initializing it with the given seed. The RandomEngine will be set to a MersenneTwister. If you wish to use a different RandomEngine with this seed, use setRandomEngine(RandomEngine). Note: Calling this method will create a new MersenneTwister. Repeated calls can lead to perfomance issues.

Parameters:
seed - - seed value
See Also:
cern.jet.random.engine.MersenneTwister