First-order discrete Markov processes
A discrete first-order Markov process is one of the simplest dynamical systems we can study. Our time variable is discrete, and we have a finite number of states between which we can move from one timepoint to the next.
Since the possible states are discrete, we can label them using integer values 1,2,3… and so on. We also use 
and 
to represent generic states.
The state at time 
is denoted by the variable 
, and so a trajectory of observations from timepoint 
to timepoint 
would be represented by the sequence 
. For example, if we have five possible states 
and we set 
, then we may have the specific trajectory 
. We can shorten the representation of the trajectory and say the observed trajectory is the sequence [3,4,4,3,4,3,5,1,2,2,4].
By taking the possible states that the system can be in to be a finite set, we simplify the mathematics considerably. However, since we can make that finite set as large as we want, we have considerable...
