The general goal of natural language processing is to uncover important information from text data.
And the main issue is that, in order to apply machine learning algorithms, the text data must be
transformed into numerical arrays. There are various ways to do this :
Bag of Words
Say we have the sentence :
A bag-of-words is a representation of text that describes the occurrence of words within a document.
We can convert the sentence into a bag-of-words representation with the following steps :
Let $s$ be the bag-of-words representation of the sentence, $s$ is defined as
$$s[i] = \text{frequency of word $i$}$$
For our sentence, we have
s = [0 3 3 2 1 1 1 1 1 1 1 1 1 1] # 0-indexed
Tf-idf
While the bag-of-words method extends natural to multiple sentences.
The tf-idf method is a more sophisticated method that is able to handle multiple sentences better, in the
sense that, it places lower weights on words that appears in multiple sentences. This allows
algorithms to focus more on words that are better suited for differentiating the sentences.
More specifically, the weights are calculated by
$$s[ d,t]=(1+\ln{f_{t, d}}) \ln \left(1+\frac{N}{1+f_{t}}\right)$$
Where
$f_{t,d}$ is the frequency of word $t$ in sentence $d$
$N$ is the number of sentences
$f_{t}$ is the number of sentences containing word $t$
This is the formula used in keras.preprocessing.text.Tokenizer with mode='tfidf'.
Other formulas may be found in different implementations, but they are all variations of the simple formula
$$f_{t, d}\frac{N}{f_t}$$
Consider, for example the following sentences :
Data science is a "concept to unify statistics, data analysis, informatics, and their related methods" in
order to "understand and analyze actual phenomena" with data.
Data science refers to the process of extracting clean information to formulate actionable insights.
The first 4 entries of Tf-Idf representations using the above formula are given by
Both Bag of Words and Tf-idf don't takes into account the ordering of words. In real life, the order
of words is important.
The n-gram method is a method that takes into account the order of words.
When $n=1$, we have the Bag of Words representation. When $n=2$, instead of having a bag of "words", we will
be getting
a bag of consecutive pair of words. For example, the following sentences : The n-grams typically are
collected from a text or speech corpus.
In our bag, we will have the following consecutive paris
For a new sentence, we will be counting how many times each consecutive pair of words appears in the sentence.
For example, the following sentence : An apple is typically collected from a tree. The 2-grams
representation will be
s = [0 0 0 0 1 1 0 0 0]
Since only the pairs
("collect
from") and ("from a") each appeared once, while all other pairs never appeared.
The extension for large $n$ is trivial.
Embeddings
When working with large text data, we are often deal with a lot of words, and when representing
sentences as vectors using what we have discussed so far, we get vectors that live in a extremely high dimensional
space.
The curse of dimensionality kicks in, and we somehow want to find a way to reduce the dimensionality of the
vectors.
Word2vec
The Word2Vec classifier that takes words to embeddings.
The idea is that, if two words are similar then their embedded vectors will be similar. We compute the similarity of
two vectors, $v,w$ , using
$$
\sigma(c \cdot w)=\frac{1}{1+\exp (-c \cdot w)}
$$Often times, we want to see how likely is a word $w$, appears with a given context of length $2L$,
$c_{-L},\cdots,c_{-1},c_{1},\cdots,c_{2L}$. This can be achieved by modified
the above formula
$$
\sigma(w;c_1,\cdots,c_L)=\prod_{i=1}^{L} \sigma\left(-c_{i} \cdot w\right)
$$
When training the Word2Vec classifier, the algorithms will train 2 embeddings (target & context) for each word, for
example, for the word "apple"
$$\text{apple}\implies \begin{cases} w_{\text{apple}}\\
c_{\text{apple}}
\end{cases}
$$
The algorithms, initialized the embeddings for each word randomly, and then iteratively update the embeddings
through
positive and negative examples.
Positive Example : Suppose "The skin of ripe apples is generally red" is an
actual sentence in the document.
Given that we are looking through a window of radius $L=3$, when come across this example in the training
process, the algorithm will bring $w_{apple}$ closer to
$$c_{skin},c_{of},c_{ripe},c_{is},c_{generally},c_{red}$$
Negative Example :
Negative examples will be generated randomly, more explicitly, we will be generating
$2L$ context words randomly, according to some distribution, one common choice is
$$\mb{P}_{0.75}(w)=\frac{count(w)^{0.75}}{\sum_{w'}count(w')^{0.75}}$$
and the algorithm will bring $w_{apple}$ further away from the context embedding of the randomly generated
context words.
Regular Expression
Move to a different page
The following table provides some commonly used regular expression building blocks.
Character Classes
escape
\
wildcard
.
digit
\d
non-digit
\D
word
\w
non-word
\W
whitespace
\s
non-whitespace
\S
new line
\n
tab
\t
Assertions
start of a line
^
end of a line
$
word boundary
\b
non-word boundary
\B
positive lookahead
x(?=y)
negative lookahead
x(?!y)
positive lookbehind
(?<=y)x
negative lookbehind
(?<!y)x
Groups and Ranges
capturing group
(x)
non-capturing group
(?:x)
or
x|y
range
[a-z] [abc]
negative range
[^a-z] [^abc]
Quantifiers
0 or more times
*
1 or more times
+
0 or 1 times
?
n times
{n}
at-least n times
{n,}
between n and m times
{n,m}
Examples
latex# I have multiple blocks begins with \textbf{Show Work:}
# and ends with \newpage
\\textbf\{Show Work:\}(\n.*?)+\\newpage
# Wants to place the content between \textbf{Show Work:}
# Dand \newpage in a hide environment
\\textbf{Show Work:}\begin{hide}$1\end{hide} \\newpage
html# find all row tags but not longrow tags
([^a]'|")row([^a]'|")
"(longrow2|longrow)"
Javascript
In Javascript, regular expressions are build-in objects.
const re = /\w+\s/g;
// words followed by a space
const str = 'fee fi fo fum';
const myArray = str.match(re);
console.log(myArray); Outputs : ["fee ", "fi ", "fo "]
// The following two expressions are equivalent
const re = /\w+\s/g;
const re = new RegExp('\\w+\\s', 'g');