scale fixed

[tsingcoo]

The output should be divided by sqrt num_units, but multiplied in your code.

[weitaizhang]

if you read the paper again , you will find the paper suggests multiply sqrt num_units by embedding weights in embedding layer.
the division happens in multi-attention layer.