My Awesome Blog

My Git Set Up

2019-11-02T00:00:00+00:00

The following is setting up for my os.

merge

keep local: grep -lr '<<<<<<<' . | xargs git checkout –ours
keep others: grep -lr '<<<<<<<' . | xargs git checkout –theirs

My os Set Up

2019-10-26T00:00:00+00:00

The following is setting up for my os.

default python to python3

1. if python3 has be installed by Homebrew, turn to 3.

Otherwise:

brew update
brew install python

2. check the location python3 has been installed

brew info python

the info is:

Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks python, python-config, pip etc. pointing to
python3, python3-config, pip3 etc., respectively, have been installed into
  /usr/local/opt/python/libexec/bin

3. edit PATH:

export PATH=/usr/local/opt/python/libexec/bin:$PATH

Note: /usr/local/opt/python/libexec/bin must be before $PATH, which will reference to python3 when excuting the command python.

python site-packages path

python -c "import site; print(site.getsitepackages())"

cmake compile json

set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

fzf.vim command “Ag”

yum install epel-release.noarch the_silver_searcher

clang

sudo yum install centos-release-scl
sudo yum install llvm-toolset-7
scl enable llvm-toolset-7 bash
clang --version

Vim Set Up

2019-10-23T00:00:00+00:00

The following is setting up for my vim.

Plugin

CurtineIncSw.vim

Toggle between .c and .h buffers.

map <C-h> :call CurtineIncSw()<CR>

fzf.vim

ctrl-t: open in a new tab
ctrl-x: a new split
ctrl-v: a new vertical split

vim-markdown

zr: reduces fold level throughout the buffer
zR: opens all folds
zm: increases fold level throughout the buffer
zM: folds everything all the way
za: open a fold your cursor is on
zA: open a fold your cursor is on recursively
zc: close a fold your cursor is on
zC: close a fold your cursor is on recursively

python-mode

zf#j creates a fold from the cursor down # lines.
zf/string creates a fold from the cursor to string .
zj moves the cursor to the next fold.
zk moves the cursor to the previous fold.
zo opens a fold at the cursor.
zO opens all folds at the cursor.
zc close a fold at the cursor.
zm increases the foldlevel by one.
zM closes all open folds.
zr decreases the foldlevel by one.
zR decreases the foldlevel to zero – all folds will be open.
zd deletes the fold at the cursor.
zE deletes all folds.
[z move to start of open fold.
]z move to end of open fold.

rtags

install: brew install rtags
start up: brew services start rtags
Start the RTags daemon: rdm&
Index the RTags project: rc -J .

Commonly used command

show variable type

[i: display the definition of a variable
[d: display a macro definition
[I: display all the lines containing the variable name
gd: go to definition of local variable (first occurrence of keyword in current function).
gD: go to definition of global variable (first occurrence of keyword in current file).

change window split size

ctrl+w +/- : increase/decrease height (ex. 20+)
ctrl+w >/<: increase/decrease width(ex. 30<C-w><)
ctrl+w _ : set height (ex. 50<C_w>_)
ctrl+w | : set width
ctrl+w = : equalize width and height of all windows
add same word for multi lines
step 1:Ctrl+v
step 2: I
step 3: write the words you want to add
step 4: ESC

tagbar for golang

Install gotags
```
go get -u github.com/jstemmer/gotags
```

Add the following to ~/.vimrc

let g:tagbar_type_go = {
  \ 'ctagstype' : 'go',
  \ 'kinds'     : [
      \ 'p:package',
      \ 'i:imports:1',
      \ 'c:constants',
      \ 'v:variables',
      \ 't:types',
      \ 'n:interfaces',
      \ 'w:fields',
      \ 'e:embedded',
      \ 'm:methods',
      \ 'r:constructor',
      \ 'f:functions'
  \ ],
  \ 'sro' : '.',
  \ 'kind2scope' : {
      \ 't' : 'ctype',
      \ 'n' : 'ntype'
  \ },
  \ 'scope2kind' : {
      \ 'ctype' : 't',
      \ 'ntype' : 'n'
  \ },
  \ 'ctagsbin'  : 'gotags',
  \ 'ctagsargs' : '-sort -silent'
\ }

ctags for golang

Add the following to ~/.ctags

--langdef=Go
--langmap=Go:.go
--regex-Go=/func([ \t]+\([^)]+\))?[ \t]+([a-zA-Z0-9_]+)/\2/d,func/
--regex-Go=/var[ \t]+([a-zA-Z_][a-zA-Z0-9_]+)/\1/d,var/
--regex-Go=/type[ \t]+([a-zA-Z_][a-zA-Z0-9_]+)/\1/d,type/`

Vanilla policy gradient

2019-10-19T00:00:00+00:00

The value-function approach on reinforcement learning has worked well in many applications. It obtains the policy by selecting the action in each state with highest estimated value iteratively.

But the approach faces several probelms. First, the policy obtained is a deterministic policy whereas the optimal policy is often stochastic in reality. Second, a little change to value fuction may result in a diffrent action is selected, which becomes a key obstacle of convergence assurances for algorithms following the value-function apporoach like Q-learning, Sarsa and so on.

Sutton,David, et al explore an alternative approach in which the policy is explicitly represented by its own function approximator, independent of the value function, and is updated according to the gradient of expected reward with respect to the policy parameters.

In reinforcement learning, the state, action, and reward are denoted $s_t \epsilon \mathit{S}$,$a_t \epsilon \mathit{A}$ and $r_t \epsilon \mathit{R}$ respectively. The environment’s dynamics are charactrized by state transition probabilities, $ P_{ss’}^{a}=P_r\left \{ s_{t+1}=s’|s_t=s,a_t=a\right \} $ and expected rewards $ R_{s}^{a}=E\left \{r_{t+1}|s_t=s,a_t=a\right \} $, $ \forall s,s’ \epsilon S, a \epsilon A $. The agent makes decision by a policy, $ \pi(s,a,\theta )=P_r(a_t=a|s_t=s,\theta) $, $ \forall s \epsilon S, a \epsilon A $, where $ \theta \epsilon \Re^l $ is a paramter vector.

There are two ways to formulate the agent’s objective with function approximation. The first is the average reward formulation. The long-term expected rewards per step, $ \rho(\pi) $ is the objective:

\[\rho(\pi)= \lim_{n \to\infty }\frac{1}{n}E\left\{r_1+r_2+....+r_n|\pi\right \}=\sum_{s}d^{\pi}(s)\sum_{a}\pi(s,a)R_s^a,\]

where $ d^{\pi}(s)=\lim_{t \to\infty}=P_r \left \{s_t=s_0, \pi \right \} $ which is the stationary distribution of states under $ \pi $.
The state-action value function is defined as:

\[Q^{\pi}(s,a)=\sum_{t=0}^{\infty}E\left \{ r_t- \rho(\pi)|s_0=s, a_0=a, \pi \right \},\forall s \epsilon S, a \epsilon A.\]

The second formulation is that in which there is a designated start state $ s_0 $. The objective $ \rho(\pi) $ is defined as:

\[\rho(\pi)=E\left \{ \sum_{t=1}^{\infty}r^{t-1}r_t|s_0, \pi \right\},\]

And the state-action value function is defined as:

\[Q^{\pi}(s,a)=E\left\{ \sum_{k=1}^{\infty}r^{k-1}r_t+k|s_t=s,a_t=a,\pi\right \},\]

where $ r \epsilon \left [0,1 \right] $ is a discount rate.

Let $ \theta $ denote the vector of policy$\pi(s,a,\theta)$ parameters and $ \rho(\pi) $ the performance of the corresponding policy, the policy gradient is:

\[\frac{\partial \rho}{\partial \theta}=\sum_{s}d^{\pi}(s)\sum_{a}\frac{\partial \pi(s,a)}{\partial \theta} Q^{\pi}(s,a) \quad \quad (1),\]

where the $d^{\pi}(s) $ has no dependence on $ \theta $.
In equation (1), $ Q_{\pi}(s,a) $ is not konwn and must be estimated. Of course, a natural thought is to use the actual returns, $ R_t=\sum_{k=1}^{\infty}r_{t+k}-\rho(\pi)$.
Now consider if $Q^{\pi} $ can be approximated by a learned function.
Assumate $ f_w:S\times A\rightarrow \Re $ is the approximation to $ Q^{\pi} $ with paramter $w$. It is natural to learn $ f_w $ by following $ \pi $ and updating $w$ by minize:

\[\left [ \hat{Q_{\pi}}(s_t,a_t) - f_w(s_t, a_t) \right ]^{2},\]

where $ \hat{Q_{\pi}}(s_t, a_t) $ is the unbiased estimator of $ Q_{\pi}(s_t, a_t)$.
When such a process has converged to a local optimum, then:

\[\sum_{s}d^{\pi}(s)\sum_{a}\pi(s,a)\left [ Q_{\pi}(s_t,a_t) - f_w(s_t, a_t) \right ]\frac{\partial f_w(s,a)}{\partial w}=0. \quad \quad (2)\]

if$f_w$ is compatible with the policy parameterization that

\[\frac{\partial f_w(s,a)}{\partial w}=\frac{\pi(s,a)}{\partial \theta} \frac{1}{\pi(s,a)}. \quad \quad (3)\]

Combine (2) and (3) then

\[\frac{\partial \rho}{\partial \theta}=\sum_{s}d^{\pi}(s)\sum_{a}\frac{\partial \pi(s,a)}{\partial \theta} f_w(s,a).\quad\quad(4)\]

The detail how to get (4) and how to proof the convergence of the policy iteration by equation (4), please see the paper (1) in the reference.

Reference

(1) Sutton, Richard S., et al. “Policy gradient methods for reinforcement learning with function approximation.” Advances in neural information processing systems. 2000.

Awesome site

2019-09-07T00:00:00+00:00

The fllowing is the papers I am reading and have read.

Computer Vision

NLP

Reinforce Learning

My paper reading list

2019-09-07T00:00:00+00:00

The fllowing is the papers I am reading and have read.

Computer Vision

NLP

Reinforce Learning

MathJax Example

2015-08-10T12:08:50+00:00

MathJax is a simple way of including Tex/LaTex/MathML based mathematics in HTML webpages. To get up and running you need to include the MathJax script in the header of your github pages page, and then write some maths. For LaTex, there are two delimiters you need to know about, one for block or displayed mathematics \[ ... \], and the other for inline mathematics $ ... $.

Usage

To enable MathJax support be sure Kramdown is your Markdown flavor of choice and MathJax is set to true in your _config.yml file.

markdown: kramdown
mathjax: true

\[a^2 + b^2 = c^2\]

Here is an example MathJax inline rendering $ 1/x^{2} $, and here is a block rendering:

\[ \frac{1}{n^{2}} \]

The only thing to look out for is the escaping of the backslash when using markdown, so the delimiters become \\[ ... \\] and \$ ... \$ for inline and block maths respectively.

\[\mathbf{X}\_{n,p} = \mathbf{A}\_{n,k} \mathbf{B}\_{k,p}\]

My Awesome Blog

My Git Set Up

merge

My os Set Up

default python to python3

1. if python3 has be installed by Homebrew, turn to 3.

2. check the location python3 has been installed

3. edit PATH:

python site-packages path

cmake compile json

fzf.vim command “Ag”

clang

Vim Set Up

Plugin

CurtineIncSw.vim

fzf.vim

vim-markdown

python-mode

rtags

Commonly used command

show variable type

change window split size

add same word for multi lines

tagbar for golang

ctags for golang

Vanilla policy gradient

Reference

Awesome site

Computer Vision

NLP

Reinforce Learning

My paper reading list

Computer Vision

NLP

Reinforce Learning

MathJax Example

Usage