This will be the first in a series on data structures and algorithms. We start with the most basic implementation of the basic BST in python, and will move onto a few more complex structures.

Binary Search Tree

The BST is, as anyone with a touch of interest in CS might know, absolutely essential. It’s the basis for most of our advanced structures for searching and sorting. It’s also the basis for many of the data structures we’ll discuss.

First, let us define what a BST is:

Definition

Let a BST be a tree (if you don’t know what a tree in CS is, you should probably figure that out before attempting advanced data structures) where each node has at most 2 children, and the left child is less than the parent, and the right child is greater than the parent.

If you know how traversals work, an inorder traversal of a BST should yield a sorted list of elements.

Use Cases

The BST is a basic data structure commonly used to search for elements. Because each level essentially divides the search space in half, the BST can be O(log n) sometimes. I say sometimes, because the tree can be O(n) very easily.

Functionality

The BST has 2 main functions, to insert and delete. Searching is so trivial to implement on a BST (check a node, if less call the search function on the left child, same for greater and right node) that we don’t really have to mention it.

Preliminaries

We’d like to create an overarching structure to use for the tree, and then implement methods to turn that tree into a BST. Since a tree can be represented as a collection of linked nodes with children, we’ll start with that.

Having a key value pair is traditional in data structures, since often times we have both a “search key” and a “value” that we want to store. Take a file structure for instance, where the name (or more specifically the path) of the file is the key, and the file content itself is the value.

python
class Node():
    def  __init__(self,
                  key        : int  = None,
                  value      : int  = None,
                  leftchild  : Node = None,
                  rightchild : Node = None):
        self.key        = key
        self.value      = value
        self.leftchild  = leftchild
        self.rightchild = rightchild

This node class is pretty simple.

Insertion

Since our BST is just a simple BST, the insert idea is also very simple. Let’s first define a template for our data structure. Ideally, we want a class that can represent all nodes in the tree. This node should have a left and right child, and a key value pair.

python
# For the tree rooted at root and the key and value given:
# Insert the key/value pair.
# The key is guaranteed to not be in the tree.
def insert(root: Node, key: int, value: int) -> Node:
    if root is None: 
        # if root is none then we are building tree
        return Node(key, value)
    cur = root # traversal setup
    while True:
        if key < cur.key:
            # if it's less and there is no left child then we can insert
            if cur.leftchild is None:
                cur.leftchild = Node(key, value)
                break
            # else recursively go into the left child
            else:
                cur = cur.leftchild
        else:
            # similar
            if cur.rightchild is None:
                cur.rightchild = Node(key, value)
                break
            else:
                cur = cur.rightchild
    return root

The insert method is pretty simple. We just traverse the tree until we find a place to insert the key. If the key is less than the current node, we go left, else we go right. If we hit a None child, we insert the key there, since we are then at a leaf. Additionally note that we include a base case, because you will find that our initializer for the tree simple includes just declaring a None node and passing in the key and value.

Deletion

Deletion is a bit more complex. Consider the following:

• If the node has no children, we can just delete it.
• If the node has one child, we can just replace the node with the child.
• If the node has two children (eg it’s a true parent node), it gets complex. Say we delete the node. How do we then “rebalance” or move around the 2 child trees so that the BST property is still maintained? It turns out that the easiest option is to find a replacement node from within the tree. Note that this option needs to be in between the left and right child of the node we are deleting.

How do we find this replacement node? We can either find the maximum of the left subtree or the minimum of the right subtree. These two are the only nodes guaranteed to be in between any of the nodes of the left tree, and any of the nodes of the right tree.

If you picture the children as sets of numbers on a line, and note that the deleted node is in the center, this may be more apparent.

In this method we use what is called the “inorder successor”, or the minimum of the right subtree. It’s called such because if you were to do an inorder traversal, this number would be immediately succeeding the deleted node.

python
# Helper function to find the inorder successor
# by finding the minimum of a tree. We will pass in
# the right subtree child of the node we are deleting.
def findminnode(node: Node) -> Node:
    cur = node
    while cur.leftchild is not None:
        cur = cur.leftchild
    return cur
# For the tree rooted at root and the key given, delete the key.
# When replacement is necessary use the inorder successor.
def delete(root: Node, key: int) -> Node:
    if root is None:
        return root # base case 
    # we look by key. key should be inorder so we use bs to find it 
    if key < root.key:
        root.leftchild = delete(root.leftchild, key)
    elif key > root.key:
        root.rightchild = delete(root.rightchild, key)
    else: # case that key=root.key 
        if root.leftchild is None: 
            return root.rightchild
        elif root.rightchild is None:
            return root.leftchild
        # inorder successor nonsense 
        # set key to smallest of right child
        tmp = findminnode(root.rightchild) # returns a node
        root.key = tmp.key
        root.value = tmp.value
        # deletus the inorder tmp!!
        root.rightchild = delete(root.rightchild, tmp.key)
    return root

Note that once we find the inorder successor, we replace the key and value of the node we are deleting with the key and value of the inorder successor. We then delete the inorder successor, which is, as it turns out, guaranteed to be either a leaf node or only have 1 child to the right (since it can’t have a left child, as it’s the smallest in that set already).

Search

Search is trivial. We just traverse the tree until we find what we want.

python
# For the tree rooted at root and the key given:
# Calculate the list of values on the path from the root down to and including the search key node.
# The key is guaranteed to be in the tree.
# Return the json.dumps of the list with indent=2.
def search(root: Node, search_key: int) -> str:
    value_list = []
    def search_helper(node: Node, search_key: int) -> bool: # im using bool to flag when to break
        if node is None: # base case
            return False
        value_list.append(node.value)
        if node.key == search_key: # we found the key and can return with the value_list complete
            return True
        if search_key < node.key:
            return search_helper(node.leftchild, search_key)
        elif search_key > node.key:
            return search_helper(node.rightchild, search_key)
    # i've been looking at the word "search" for too long and now it looks stupid
    search_helper(root, search_key)
    return value_list

And we have a simple search function that returns the path to the key.

Closing Thoughts

This is a simple implementation of a BST. It’s not the most efficient, but it’s a good starting point. We can improve this by balancing the tree, which we will discuss in a future post. I hope you learned something; although hopefully the concepts were already familiar to you, since a BST is by far the most basic structure I intend to go over. Also, if I feel good after finishing this series, I may start one on binary exploitation.

Going to aim to keep this introduction as simple as possible. It’s heavily inspired by Aris Papadopoulos’ course Math445, which I took last semester.

I also want to mention that, yeah, I’ve been dead. I have an unlisted advanced datastructures series that I want to finish, and additionally I need to keep working on the machine learning stuff. I also want to start a new one on security. But for now, logic.

Assumptions

I will be assuming that you have some basic familiarity with sets, functions, and proofs. If you don’t, there are a lot of resources online that can help you get up to speed.

A “section” will be a self contained part of this article. Each section is designed to contain its own definitions and theorems, and new sections may depend on previous sections, but not vice versa. Some definitions might not pass through either.

Math

Basic Definitions

Definition (Equinumerous): Two sets $X$ and $Y$ are equinumerous if there exists a bijection (a one-to-one and onto function) $f: X \to Y$. We can write this as $X \sim Y$.

The gist of this is that there are an equal “number” of elements in both sets. If this is true, then naturally there is a function that maps each element to each other using the index of the element. The definition of “number” is dubious of course, since we know that $\mathbb{N} \sim \mathbb{Z} \sim \mathbb{Q}$, and we’ll get into this soon.

Definition (Subnumerous): $X$ is subnumerous to $Y$ if there exists an injection (a one-to-one function) $f: X \to Y$. We can write this as $X \preceq Y$. The gist of this is that there are “less than or equal to” elements in $X$ as there are in $Y$.

Definition (Power Set): The power set of a set $X$, denoted $\mathcal{P}(X)$, is the set of all subsets of $X$.

Theorem (Cantor’s Theorem)

For any set, $A$, $A$ is not equinumerous to $\mathcal{P}(A)$. In other words, there is no bijection between a set and its power set. Here, the power set is defined as above.

Proof #1: This proof, more overarching, does not use the diagonal argument.

Take an arbitrary map $f$ from $A$ to $\mathcal{P}(A)$. We will show that this map cannot be surjective, and thus cannot be a bijection. To prove this, we will construct a set that is not in the image of $f$ (since the definition of surjective is that every element in the codomain has a preimage in the domain).

Then, let $B = \{ x \in A \mid x \not\in f(x) \}$. In other words, $B$ is the set of all elements in $A$ that are not in their own image under $f$. Note that $B$ is in the power set of $A$, since it is a subset of $A$ (we are only taking elements from $A$ to form $B$).

Then, $B=f(b)$ for some $b \in A$ if and only if $b \in B$ if and only if $b \not\in f(b)$ if and only if $b \not\in B$. This is a contradiction, so there is no such $b$, and thus $B$ is not in the image of $f$. Therefore, $f$ is not surjective, and thus cannot be a bijection.

However, this proof can be a little bit unsatisfying, since it doesn’t really give us a concrete example of a set that is not in the image of $f$. In fact, we define some set $B$ that depends on $f$, so it feels a bit like cheating.

Proof #2 (for $\mathbb{N}$): Now, we will use the diagonal argument to construct a concrete example of a set that is not in the image of $f$, for the case where $A=\mathbb{N}$.

Take any arbitrary map $f$ from $A$ to $\mathcal{P}(A)$. For the sake of simplicity and without loss of generality let

etc. Note that the domain is elements of $A$ (which for now we’ll take to be $\mathbb{N}$ for simplicity), and the range is subsets of $A$.

Then, you can represent this as a matrix of 0s and 1s, where each row represents the image of an element in $A$, and each column represents whether or not an element is in the subset. For example, for $f(1), f(2)$, we have that

Then, for each diagonal element (i.e. the elements where the row and column indices are the same), we flip the value (0 becomes 1, and 1 becomes 0). This gives us a new set of values that differ from each row in at least one position, always. Thus, $B=\{ x \in A \mid \text{the } x\text{-th diagonal element is } 1 \}$ is not in the image of $f$, since it differs from each $f(x)$ in at least one position (the $x$-th position).

That is, $B$ contains $1$ if $1 \not\in f(1)$, contains $2$ if $2 \not\in f(2)$, and so on. Thus, $B$ cannot be equal to any $f(x)$. Note that we must use the diagonal here because if we just flipped the first element of each row, for example, then it is possible that two rows could be identical in all other positions, and thus we wouldn’t be able to guarantee that $B$ is not in the image of $f$ because we’ve inverted them into each other.

Exercise: It’s an extremely interesting (and kind of annoying, if you want to do it “correctly”) exercise to prove that $\mathbb{N}\sim \{0,1\}^\mathbb{N}$ (the set of all infinite binary sequences). You can also do this for $\mathbb{N}\sim \mathbb{Q}$ if you want to go a bit further, but that’s further more annoying.

Finishing Remarks

Definition (Countable): A set $X$ is countable if it is equinumerous to $\mathbb{N}$ (or finite). Otherwise, it is uncountable.

For instance, $\mathbb{N}, \mathbb{Z}, \mathbb{Q}$ are countable, while $\mathbb{R}$ is uncountable.

Note: One interesting fun note is that the continuum hypothesis, which states that there exists no set $X$ such that $\mathbb{N} \prec X \prec \mathbb{R}$. In other words, there is no set with cardinality strictly between that of the integers and the real numbers. This was shown to be independent of the standard axioms of set theory (ZFC, or the Zermelo-Fraenkel set theory with the axiom of choice), meaning that it can neither be proven nor disproven using those axioms. That is to say, both the continuum hypothesis and its negation are consistent with ZFC, assuming ZFC itself is consistent.

That is to say, ZFC is not capable of deciding the truth value of the continuum hypothesis within its own system. This tells us that a seemingly good set of axioms can still be incomplete in some sense. We’ll get into this more later!

For fun, you may want to read up on ZFC here.

End

Next: Propositional logic

Propositional Logic is a deep topic, covering most of what we can call “logical reasoning”. You have probably seen most of it before, and the goal today is to use it towards proving some kind of completeness theorem.

Be warned that at the moment, I have no intention of putting proofs in here. The “introduction” posts will be definitions only, pretty much. I may come back later and have a series of posts titled eg. “Proofs of Propositional Logic”, but for now, just definitions.

Math

The general idea is that propositional logic allows us to define a set of propositions, and then use logical connectives to build more complex propositions. We will distinguish between syntax, semantics, and the proof system that ties them together.

Basic Definitions

A propositional logic system must be able to express some kind of statement, give a meaning to that statement, and then have a way to prove things about those statements.

To put it another way, think of the typical Algebra 1 expressions you work with. We need a way to write expressions and formulas, eg “x+y” and “2+3”, a way to assign meaning to those expressions, eg “x=2” and “y=3”, and then a way to prove things about those expressions, eg “2+3=5”.

Structure of Propositional Logic

Definition (Propositional Symbols): I include the tex here for easy copy pasting for those using tex.

Note that we are defining grouping here!

If you have ever taken a programming languages course, you will know that grouping is very important for defining order of operations. We will not define a “parser” here, but know that parentheses are very important, and that when we evaluate expressions, we will always evaluate it as if there was a parser.

For instance, we will always evaluate the innermost parentheses first. Additionally, we must group every expression with parentheses. For example, instead of writing $A \wedge B \rightarrow C$, we will write $((A \wedge B) \rightarrow C)$.

If I do not write parentheses, it should be obvious from context that I am omitting them for brevity, and that it can be grouped in a way that makes sense.

Definition (Propositional Variable): A propositional variable, the set of which will be denoted $Var$, is just some set of symbols that represent a proposition. For example, $P$, $Q$, and $R$ could be propositional variables. The cardinality of this set must be at least countably infinite, eg $|Var|=\aleph_0$.

Definition (Propositional Formula): A propositional formula is the following:

• Any propositional variable is a propositional formula.
• Propositional formulas are closed under the logical connectives defined above. For example, if $A$ and $B$ are propositional formulas, then so are $(A \wedge B)$, $(A \vee B)$, $(\neg A)$, and $((A) \rightarrow (B))$.
• Nothing else is a propositional formula.

Note that this is a recursive (or inductive) definition. We can build up complex formulas from simpler ones, and that is the only way we can do them. It also means we can break down complex formulas into simpler ones by finding a recursive path that builds it up, and then reversing that path. Note that this recursive path also doesn’t have to be unique under valuation; for example, $((A \wedge B) \vee C)$ can be built up in two different ways depending on which connective we apply first, which are $((A \wedge B))$ and $C$, or $A$ and $(B \vee C)$.

When discussing how to build a tree of a propositional formula, we may use “recursive” because it’s more computational. However, when discussing proofs by induction on propositional formulas, we will use “inductive” since it’s more mathematical.

TLDR: a propositional formula is the smallest possible set of things that contains all propositional variables and is closed under the logical connectives.

Because formulas are inductive items, proofs on them will be done with structural induction, eg by proving a base case and then showing that the property holds under closure.

Definition (Subformula): The function $Sub$, which defines the set of subformulas of a propositional formula, is defined as follows:

• $Sub(P) = \{ P \}$ for any propositional variable $P$.
• $Sub((\neg A)) = \{ (\neg A) \} \cup Sub(A)$.
• $Sub((A \wedge B)) = \{ (A \wedge B) \} \cup Sub(A) \cup Sub(B)$.
• $Sub((A \vee B)) = \{ (A \vee B) \} \cup Sub(A) \cup Sub(B)$.
• $Sub((A \rightarrow B)) = \{ (A \rightarrow B) \} \cup Sub(A) \cup Sub(B)$.
• Nothing else.

As an example, let’s prove that for all formulas, $\phi$, we have that $\phi \in Sub(\phi)$. Again, we will do this by structural induction.

Example: We will prove that for all formulas, $\phi$, we have that $\phi \in Sub(\phi)$.

Proof: We will do this by structural induction on $\phi$.

• Base Case: Let $\phi$ be a propositional variable, $P$. Then, by the definition of $Sub$, we have that $Sub(P) = \{ P \}$, so $P \in Sub(P)$.
• Inductive Step: Assume that for all formulas $A$ and $B$, we have that $A \in Sub(A)$ and $B \in Sub(B)$. We will show that this holds for the formulas $(\neg A)$, $(A \wedge B)$, $(A \vee B)$, and $(A \rightarrow B)$.
  • Let $\phi = (\neg A)$. Then, by the definition of $Sub$, we have that $Sub((\neg A)) = \{ (\neg A) \} \cup Sub(A)$. By the inductive hypothesis, we have that $A \in Sub(A)$, so $(\neg A) \in Sub((\neg A))$.
  • Let $\phi = (A \wedge B)$. Then, by the definition of $Sub$, we have that $Sub((A \wedge B)) = \{ (A \wedge B) \} \cup Sub(A) \cup Sub(B)$. By the inductive hypothesis, we have that $A \in Sub(A)$ and $B \in Sub(B)$, so $(A \wedge B) \in Sub((A \wedge B))$.
  • Let $\phi = (A \vee B)$. Then, by the definition of $Sub$, we have that $Sub((A \vee B)) = \{ (A \vee B) \} \cup Sub(A) \cup Sub(B)$. By the inductive hypothesis, we have that $A \in Sub(A)$ and $B \in Sub(B)$, so $(A \vee B) \in Sub((A \vee B))$.
  • Let $\phi = (A \rightarrow B)$. Then, by the definition of $Sub$, we have that $Sub((A \rightarrow B)) = \{ (A \rightarrow B) \} \cup Sub(A) \cup Sub(B)$. By the inductive hypothesis, we have that $A \in Sub(A)$ and $B \in Sub(B)$, so $(A \rightarrow B) \in Sub((A \rightarrow B))$.

Thus, by structural induction, we have that for all formulas $\phi$, we have that $\phi \in Sub(\phi)$.

This is a very bulky proof! For the future, I will truncate similar proofs to just the base case and one inductive step, since the rest are analogous. Additionally, we will avoid trying to prove things that are obvious.

Definition (Variables): Let’s formally define the function $Var(\phi)$, which gives us the set of propositional variables in a formula $\phi$. Note that $Var()$ is different than $Var$. The first is a function that produces a set of propositional variables from a formula, while the second is the set of all propositional variables.

• $Var(P) = \{ P \}$ for any propositional variable $P$.
• $Var((\neg A)) = Var(A)$.
• $Var((A \wedge B)) = Var(A) \cup Var(B)$.
• … and so on for the other connectives.
• Nothing else.

Substitution

The notion of substitution is something that we should all understand to some degree. In 6th grade algebra, when we have the expression $x + 2$, and we don’t know what $x$ is in this expression, but we do know that $x = 3$ from some other context, we can replace $x$ with $3$ to get $3 + 2 = 5$.

The idea is that we can replace things in propositional logic too. They can be from within the same universe of discourse, or from different ones. For example, if we have the formula $(P \wedge Q)$, and we know that $P$ is equivalent to $(R \vee S)$, then we can replace $P$ with $(R \vee S)$ to get $((R \vee S) \wedge Q)$.

Definition (Substitution): Let $A\in Var$ (recall that this is the set of all propositional variables), and let $\phi$ be a propositional formula. Let $\psi$ be another propositional formula. Then, we define the substitution of $A$ with $\psi$ in $\phi$ as

These two notations are equivalent. We will use the first. In order for it to be well defined, let’s have that

• For $B\in Var$, then $B[\psi/A] = \psi$ if $B = A$, $B$ if $B \neq A$
• For $(\neg \phi)$, then $(\neg \phi)[\psi/A] = \neg (\phi[\psi/A])$
• For $(\phi_1 \wedge \phi_2)$, then $(\phi_1 \wedge \phi_2)[\psi/A] = (\phi_1[\psi/A] \wedge \phi_2[\psi/A])$
• … and so on for the other connectives.
• Nothing else.

A shorthand we’re going to make here is that

Next, we will define a lemma, which basically just states that substitution on a variable that does not exist in the formula does nothing. To make an analogy if $x=5$ and we have the expression $y + 2$, substituting $x$ with $5$ does nothing since $x$ is not in the expression.

Lemma: If $A\notin Vars$, then $\phi[\psi/A]=\phi$

Proof: We will use induction on the structure of formulas.

Base case: Let $\phi=B\in Var$. Then, assuming that $A\notin Vars(B)$, eg that $A\neq B$, then $\phi[\psi/A]=B$.

Inductive Step: Assume that for formulas $\psi, \phi$ if $A\notin Vars(\phi)$ then $\phi[\psi/A]=\phi$, and if $A\notin Vars(\psi)$, then $\psi[\psi/A]=\psi$. We must show it holds for $\neg, \wedge, \vee, \rightarrow$.

• For $(\neg \phi)$, if $A\notin Vars((\neg \phi))$, then $A\notin Vars(\phi)$. Thus, by the inductive hypothesis, we have that $\phi[\psi/A]=\phi$. Therefore, $(\neg \phi)[\psi/A] = \neg (\phi[\psi/A]) = \neg \phi$.
• For $(\phi_1 \wedge \phi_2)$, if $A\notin Vars((\phi_1 \wedge \phi_2))$, then $A\notin Vars(\phi_1)$ and $A\notin Vars(\phi_2)$. Thus, by the inductive hypothesis, we have that $\phi_1[\psi/A]=\phi_1$ and $\phi_2[\psi/A]=\phi_2$. Therefore, $(\phi_1 \wedge \phi_2)[\psi/A] = (\phi_1[\psi/A] \wedge \phi_2[\psi/A]) = (\phi_1 \wedge \phi_2)$.
• … and so on for the other connectives.

Definition (Propositional Sentence): A propositional sentence is a propositional formula with no free variables. In other words, all variables in the formula are bound by some quantifier. At the moment, since we have not defined quantifiers, all propositional formulas are propositional sentences.

Tautologies, Contradiction, Entailment, and Proofs

Now that we have some basic definitions and syntax out of the way, we want to define some semantics. The idea is that we want to be able to assign truth values to propositional formulas, and then use those truth values to determine whether a formula is a tautology, contradiction, or contingent.

What does truth mean?

Definition (Truth Table): We want to give meetings to our formulas, and one of the easiest ways to do that is via truth tables. A truth table represents a truth function, which is a mapping from propositional variables to truth values (True or False).

We will denote $T$, or $True$, as the truth value “true”, and $F$, or $False$, as the truth value “false”. Propositional variables can take on either truth value.

Then, truth functions map ${F,T}^{|S|} \to {F,T}$, where $S$ is some set of propositional variables. For example, the truth function for $(P \wedge Q)$ is defined as follows:

If you are unfamiliar with the basic definitions of our connectives, here is a quick refresher: the conjunction $(P \wedge Q)$ is true if both $P$ and $Q$ are true, and false otherwise. The disjunction $(P \vee Q)$ is true if at least one of $P$ or $Q$ is true, and false otherwise. The negation $(\neg P)$ is true if $P$ is false, and false if $P$ is true. The implication $(P \rightarrow Q)$ is false only when $P$ is true and $Q$ is false; otherwise, it is true. These can be summarized in truth tables as well.

Note that we have ex falso quodlibet here: from a contradiction, anything follows. That is to say, $F \rightarrow x$ is always true, regardless of the truth value of $x$.

Definition (Truth Assignment): A truth assignment is a function $\mathcal{A}$ which assigns each propositional variable a truth value. That is, $\mathcal{A}: Var \to \{T,F\}$. This gives a meaning to propositional formulas.

Then, we can define the evaluation of a propositional formula under a truth assignment! Assume that we have, defined by truth tables, the functions $f_{\wedge}, f_{\vee}, f_{\neg}, f_{\rightarrow}$ which give the truth values of the respective connectives given their inputs.

Definition (Valuation Function): Then, we can define the valuation function $\mathtt{val}$ as

• $\mathtt{val}_{\mathcal{A}}(P) = \mathcal{A}(P)$ for any propositional variable $P$, eg $P \in Var$.
• Let $\phi_1$ and $\phi_2$ be propositional sentences (a reminder that in our current system a propositional sentence is equivalent to a propositional formula). Then,
  • $\mathtt{val}_{\mathcal{A}}((\neg \phi_1)) = f_{\neg}(\mathtt{val}_{\mathcal{A}}(\phi_1))$
  • $\mathtt{val}_{\mathcal{A}}((\phi_1 \wedge \phi_2)) = f_{\wedge}(\mathtt{val}_{\mathcal{A}}(\phi_1), \mathtt{val}_{\mathcal{A}}(\phi_2))$
  • … and so on for the other connectives.

We write $\phi[\mathcal{A}]$ as shorthand for $\mathtt{val}_{\mathcal{A}}(\phi)$.

Definition (Models): We say that a truth assignment $\mathcal{A}$ is a model of a propositional formula $\phi$ if $\mathtt{val}_{\mathcal{A}}(\phi) = T$. Then, we write that $\mathcal{A} \models \phi$. In other words, the formula evaluates to true under the truth assignment. Similarly $\mathcal{A}\not\models \phi$ if $\mathtt{val}_{\mathcal{A}}(\phi) = F$.

Definition (Tautology): A formula $\phi$ is a tautology if for ALL assignments $\mathcal{A}$, we have that $\mathcal{A} \models \phi$. In other words, the formula is always true regardless of the truth values of its propositional variables. We denote this as $\models \phi$.

An example of a tautology is $(P \vee \neg P)$, which is true regardless of whether $P$ is true or false. (you can choose any valuation for $P$, and it will always be true)

This tautology is called the law of excluded middle. Another tautology is MP, or Modus Ponens, which states that $((A\wedge (A \rightarrow B)) \rightarrow B)$ is a tautology. That is to say, if we know $A$, and $A\to B$, then we can conclude $B$.

It’s somewhat controversial if the law of excluded middle should be a tautology, since in some logics (eg intuitionistic logic), it is not. However, in classical propositional logic, it is. And MP is definitely a tautology in all standard propositional logics.

Definition (Logical Entailment): A formula $\phi$ logically entails a formula $\psi$ if for all $\mathcal{A}$, such that $\phi[\mathcal{A}]=T$, we have that $\psi[\mathcal{A}]=T$. In other words, whenever $\phi$ is true, $\psi$ is also true. We denote this as $\phi \models \psi$.

More generally, for a set of formulas $\Gamma$, we say that $\Gamma$ logically entails $\psi$ if for all $\mathcal{A}$, such that for all $\phi \in \Gamma$, we have that $\phi[\mathcal{A}]=T$, then $\psi[\mathcal{A}]=T$. In other words, whenever all formulas in $\Gamma$ are true, then $\psi$ is also true. We denote this as $\Gamma \models \psi$.

Definition (Logical Equivalence): We say that a formula $\phi$ is logically equivalent to a formula $\psi$ if $\phi\leftrightarrow \psi$ is a tautology. In terms of logical entailment, this is equivalent to saying that $\phi \models \psi$ and $\psi \models \phi$. We denote this as $\phi \equiv \psi$.

It’s a fun exercise to prove that if $\phi$ and $\psi$ are formulas, then the following are equivalent: $\phi$ logically implies $\psi$, and $(\phi \rightarrow \psi)$ is a tautology. In symbols, $\phi \models \psi$ if and only if $\models (\phi \rightarrow \psi)$.

Definition (Unsatisfiable): A formula $\phi$ is unsatisfiable if there is NO truth assignment $\mathcal{A}$ such that $\mathcal{A} \models \phi$. In other words, the formula is always false regardless of the truth values of its propositional variables. We can also say that a set of formulas $\Gamma$ is unsatisfiable if there is NO truth assignment $\mathcal{A}$ such that for all $\phi \in \Gamma$, we have that $\mathcal{A} \models \phi$.

Truth Functions

Let $\mathbb{A}$ be the set of all truth assignments. Then, we should intuitively be able to define a function from truth assignments to truth values for any propositional formula. This is called a truth function.

Definition (Truth Function): A function $f: \mathbb{A}\to \{T,F\}$ is a truth function if there exists a finite set of variables $\{A_1,\dots, A_n\}\subseteq Var$ such that for all assignments $\mathcal{A}, \mathcal{B}$ we have that if

Then, we have that $f(\mathcal{A}) = f(\mathcal{B})$. In this case, we can call $\{A_1,\dots, A_n\}$ the set of variables that $f$ depends on, or the support of $f$.

Then, given some formula $\phi$ we can define the function for $\phi$ in the most obvious way:

Note that every propositional variable implies a truth function, where

The support of this function is just $\{P\}$.

Additionally, all truth functions can be described by a finite truth table. This is because the support of a truth function is finite, and thus there are only finitely many combinations of truth values for the variables in the support. Thus, we can enumerate all possible combinations of truth values for the variables in the support, and then define the output of the function for each combination. This is of course impractical for large supports, but it is theoretically possible, and might remind you that we are working with finite, countable objects here.

Definition (Adequacy of Truth Functions): Let $\mathcal{F}$ be a set of truth functions. We will say that $\mathcal{F}$ is adequate if for every truth function $f$, there exists a formula $\phi$ such that the truth function induced by $\phi$ is equal to $f$, and $\phi$ only uses truth functions from $\mathcal{F}$.

That is, every truth function $g:\{T,F\}^n \to \{T,F\}$ can be expressed as a propositional formula using only the truth functions in $\mathcal{F}$ under composition.

Theorem: I don’t really do theorems in this because I didn’t want to bloat the post with proofs, but it’s worth stating that the set of truth functions $\{\wedge, \vee, \neg\}$ is adequate. It might be a fun exercise to show that the remaining functions $\rightarrow$ and $\leftrightarrow$ can be expressed in terms of these three functions.

From this, you can also deduce that $\{\rightarrow, \neg\}$ is adequate, since you can express $\wedge$ and $\vee$ in terms of these two functions.

End

Next: Propositional logic proof systems.

There’s a lot. like, a LOT. of math, and history, that we can go into with regards to neural networks. For the sake of trying to make the article more approachable (if you’re using my articles as a reference paper for your classes you REALLY shouldn’t) I’m going to gloss over anything that’s not essentially high school maths. Let it be known that I will not be directly referencing many of the founding papers of neural networks.

If you’re interested about the specifics of neural models with regards to LLMs, try reading this paper by Bengio et al. And for those who want to approach the math of backprop in more detail, read this blog post by Karpathy!

Learning Basics

Neural networks can easily be seen as a black box that takes in some input, and spits out some output. Okay, great. For the sake of introduction, suppose that it is, in fact, a black box. Let’s create a basic 1 node network that will learn to give us y=x.

The node starts off by operating on one input (a number), and gives one output (also a number). The node has a weight, which is a number that it multiplies the input by. The node also has a bias, which is a number that it adds to the output. These are both randomly initialized.

Using python, our node looks like this:

python
class Node:
    def __init__(self):
        self.weight = random.random()
        self.bias = random.random()
    def forward(self, x):
        return x * self.weight + self.bias

Note that we are using the forward method to compute the output of the node. This is because we will later add more methods to the node, and we want to keep the code organized.

Adjusting for Loss

Now, we want it to give us y=x, the “target”. We will do this by using a loss function, which will figure out how far away the output is from the “target” and then update the weights and biases accordingly.

python
class Node:
    def __init__(self):
        self.weight = random.random()
        self.bias = random.random()
    def forward(self, x):
        return x * self.weight + self.bias
    def loss(self, x, y):
        output = self.forward(x)
        # Note that we square this here to make sure that the loss is always positive
        # This is called the Mean Squared Error (MSE) loss
        return (output - y) ** 2

Now that we have this loss, how do we update the weights and biases? We will use a method called gradient descent. This is a method that will update the weights and biases in the direction that will minimize the loss. This goes into the math of derivatives, but the essence is that we figure out how much the loss will change if we change the weight or bias by a small amount, and then update the weight or bias in the direction that will minimize the loss.

We calculate the gradient as

where $L$ is the loss, $w$ is the weight, $N$ is the number of samples, $y_i$ is the target, $\hat{y}_i$ is the output, and $x_i$ is the input.

This is the gradient of, specifically, the MSE loss function. MSE is commonly used for regression problems (like this one), where we want to have a common loss function that is easy to compute and differentiable.

Much more generally, gradient can be calculated as

where $\hat{y}_i$ is the output of the model, and $w$ is the weight. This is called the chain rule, and it allows us to compute the gradient of any loss function with respect to any weight.

Then, we update the weights and biases as

Where $\alpha$ is the learning rate, which is a hyperparameter that controls how much we update the weights and biases. Hyperparameters are parameters that are not learned (like weights and biases are, for us) but are set before training.

Generalizing this again, we could imagine that a node with many weights and many biases can be computed as

where $f$ is some function (we’re going to go over this later, but if you’re familiar with activation functions, this is that), and $w_i$ and $b_i$ are the weights and biases. For now note we aren’t using any activation function, so $f$ is just the identity function.

Training

Now we have all the pieces, we can “train” our node by giving it a bunch of inputs and the corresponding targets (we call this the “dataset”), and then updating the weights and biases using gradient descent.

Essentially, since we already know y=x, we can just give it a bunch of input/output pairs, and if we define the gradient descent correctly it will learn to give us y=x.

python
class Node:
    def __init__(self):
        self.weight = random.random()
        self.bias = random.random()
    def forward(self, x):
        return x * self.weight + self.bias
    def loss(self, x, y):
        output = self.forward(x)
        return (output - y) ** 2
    def gradient(self, x, y):
        output = self.forward(x)
        dL_dy = 2 * (output - y)
        dy_dw = x
        return dL_dy * dy_dw
    def update(self, x, y, alpha):
        grad = self.gradient(x, y)
        self.weight -= alpha * grad
    def train(self, dataset, alpha):
        for x, y in dataset:
            self.update(x, y, alpha)
node = Node()
dataset = [(i, i) for i in range(100)]
node.train(dataset, 0.01)

Note that we never actually defined the function y=x anywhere in here! We just gave it a bunch of random numbers and it was able to “correct” itself to learn the function.

Imagine you’re blind, and you have a ball in your hand. You want to throw the ball to your friend, but you can’t see them. First, you throw the ball. It likely won’t hit your friend, but you have an “oracle” that tells you how far off you were. You then adjust your throw based on the feedback, and throw again. You keep doing this until you hit your friend. This is essentially what we did here.

Neural Networks

Well, neural networks are just a bunch of nodes. Imagine that instead of one we had, let’s say, 10 nodes, 100 nodes, or (as many LLMs do nowadays), tens of billions of nodes.

Structure

How do we connect these nodes? Well, we can connect them in a few ways. The most common way is to have a “layer” of nodes, where each node is connected to every other node in the next layer. This is called a fully connected layer.

There are also other types of layers, such as convolutional layers, which are used for image processing, and recurrent layers, which are used for sequence processing.

We can also have “skip connections”, where we connect nodes from one layer to nodes in a later layer. This is called a residual connection.

Other types of model architectures include transformers, which are used for sequence processing, and graph neural networks, which are used for graph processing.

Many of these are not very relevant for LLMs and language modelling, so we won’t go too hard into them in this series. However, I highly suggest going over them if you’re interested in the field of machine learning!

Activation Functions

We were talking about that function $f$ earlier. This is called an activation function, and it is used to introduce non-linearity into the model.

Essentially, if you didn’t have any activation functions, then the nodes (or neurons), could just “collapse”. To illustrate, suppose we had 2 nodes as simple as the one earlier, in series. Then,

and

Then, you could expand this to

Note that this is just a linear function $ax+b$ where $a=w_2w_1$ and $b=w_2b_1+b_2$.

This means that if you had a network of 1000 nodes, you could just collapse it into one node with a weight of $w_1w_2\dots w_{1000}$ and a bias of $b_1+b_2+\dots+b_{1000}$.

Thus, we want “nonlinearities” in between in order to allow for complex behavior.

There are many activation functions, but the most common ones are:

• ReLU: $f(x) = \max(0, x)$
• Sigmoid: $f(x) = \frac{1}{1+e^{-x}}$
• Tanh: $f(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$
• Softmax: $f(x_i) = \frac{e^{x_i}}{\sum_{j=1}^n e^{x_j}}$

There’s a whole wikipedia article with a table of other ones if you’re interested!

The most common one is the ReLU, which is used in most modern neural networks. The sigmoid and tanh are used in older neural networks, and the softmax is used in the output layer of classification models.

There’s a lot of reasons why you might use one over the other (and softmax in particular is only really used in the output layer of non-regression models), but it actually turns out that with enough compute and data you can pretty much use any activation function and get similar results.

Fin

Well, that’s it for this one, just a quickie. This basic setup of nodes allows you to “learn” and regress to very basic functions, like polynomial math equations.

In the next one we’ll go over a bit of intermediary knowledge and then jump to RNNs, which is where LLMs really started to become powerful.

So we have our N-Gram model (see N-Gram Model Prediction in Machine Learning and LMs). Now, how do we evaluate it? And why do we need to evaluate it?

Well, to make any progress in the field we would like to be able to compare new models. Thus, we need some kind of metric to compare models. We now set out to find methods to evaluate models, and then discuss training and testing data.

First, some definitions;

• A training set is the set of data we use to learn the parameters of our model. This becomes much more relevant in the future when we start discussing neural models, but it’s essentially the data that we use to “train” our model.
• A test set is a completely different set of data from the training set. It will be kept completely witheld from the model until it comes time for testing, and we will use it to evaluate the model’s performance. We want this set to be diverse, and sometimes even separate entirely by context from the training set, since in the context of NLP the ability to generalize is a huge part of the model’s performance. It importantly must be separate because we want to see how well the model generalizes to new data, not how well it memorizes the training data.
• The dev set, sometimes called the validation set, is a set that we use in neural models to tune parameters. This may not be entirely relevant now, but will be later; this is what we use to do testing during training. Sometimes, you want feedback before the model is done; this is what we use the devset for. It’s important that the devset is correlated to the test set in context but do not overlap.

Evaluation

The most obvious idea to evaluate the performance of an LM is extrinsic evaluation. This idea essentially is to take the model, put it in a real-world scenario, and see how well it performs. For instance, we could use the model to generate text, and then have humans rate the text. Alternatively, for something like a translation model or speech recognition, we might want to run two comparable models and then choose the better outcome.

However, running huge NLP models with this level of complexity is both expensive and time consuming; not to mention that the best judge of this sort of thing would be a human, which we don’t like involving in training processes. Thus, we would like to have some kind of intrinsic evaluation, or something that can quickly evaluate the quality of a model without any application

Perplexity

The perplexity of a test set is the inverse probability of the test set, normalized by the number of words. That is, we’re looking for the probability that the model will predict the test set. Hopefully that’s intuitive; the best way to evaluate a model internally is to see how well it will predict some data that it has never seen before that mirrors real life.

We can’t use the test set for mid-training evaluation, for which we use the dev set.

You can normalize by the number of words $N$ by taking the $N$th root of the perplexity value. For some test set of words $W=w_1, w_2, \ldots, w_N$, the perplexity is defined as

I encourage you to take some time to convince yourself of where this came from. To briefly demonstrate why, the idea is to take $P(w_1, w_2, \ldots, w_N)$ and then take the $N$th root of it. After expanding with the chain rule we have the above.

Note because of the inversion, the higher the probability of the word sequence, the lower the perplexity will be. Consequently, we will train for a lower perplexity value.

In other words, the lower the perplexity value, the higher the probability that our trained model will predict the completions in our dev/test sets.

Weighted Average Branching Factor

Another way to think about perplexity is to imagine it as the so called “weighted average branching factor”. A branching factor is simply the number of branches, or possibilities, that follow any word.

For instance, consider a situation that only considers a vocabulary of 3. Then, the branching factor is 3. Call these words $w_1, w_2, w_3$. Then, assume that we have a LM N-Gram model that predicts the next word with all 3 being equally likely with a chance of $1/3$. Then, suppose we have a test set with exactly 1 entry, of $w_1 w_1 w_1 w_1 w_2$.

First, the perplexity of this model is demonstrably 3. That is,

Now consider an N-gram model, called 2, that works in the same context with the same vocabulary but was trained to predict $P(w_1)=.8, P(w_2)=.1, P(w_3)=.1$. Then, the perplexity of this model is demonstrably lower.

(A brief summary on) Improvements of Performance

N-Gram models are still very problematic. Here I give a brief overview of some methods used to improve performance. These have all been advanced very far in the field by amazing researchers, and this stuff goes back decades. I don’t intend to cover things in depth here, but here’s a general idea of a lot of the things and I encourage you to read the literature if you’re interested in N-Grams or NLP in general.

Sampling

When you predict the next word in a N-Gram model, the obvious method is simply to choose the most probable result. However, we can use sampling instead to choose less likely answers.

Intuitively, when you speak, you don’t always use the most common word. As an example, if it were raining outside, you might not always say “It is raining”. You might say “It is lightly raining”, or “It is pouring”. These sentences would never be completed by a model that always chooses the most probable word, because “lighly raining” and “pouring” are (at least in the year im writing this, haha) much less likely to appear in a training set than simply “raining”.

Consequently, we want there to be some chance of choosing a less likely word. This is called sampling.

One intuitive idea is to choose each word with a probability proportional to its likelihood. That is, if we have a word $w$ with probability $P(w)$, then we can choose $w$ with probability $P(w)$. This is called unnormalized sampling, and can help your model to be more diverse.

There’s a lot of sampling methods! And a lot of papers on them! I encourage the reader to check them out with a search on N-Gram sampling methods.

Training Sets

Just like neural models, which we will go over later, the N-Gram model is also very dependent on the training set and can easily overfit. That is, it can “memorize” or get used to things too easily.

Suppose that I gave the N-Gram model exclusively sentences that appear in Jane Austen’s works. I regret to inform the reader that I’m not really a fan of Austen’s older English, and that seeing “thou” and “thee” in every sentence would be quite annoying, and certainly not representative of a modern English speaker. Consequently, our N-Gram model can be said to be “overfitted” to the training set.

We thus require a diverse training set, that doesn’t just include one specific context (for instance we can’t just have school conversations because they won’t have any information on how to talk about the weather or trashtalk people in Call of Duty). Yet similarly, we want to restrict our training data to the appropriate variety that we want to include. Maybe you actually don’t want to include Call of Duty trashtalk in your model, and that’s fine too.

Smoothing

A problem with using the MLE estimate for probability is that some completely fine English sentences will have a probability of 0. This is because we can’t possibly have seen every single sentence in the English language, and so some sentences will have a probability of 0.

An example? “Green Eggs”. You would never expect to see those two words together in a lot of datasets, and you’d rarely hear about it irl, but despite that it’s, well, definitely valid and is part of Dr Seuss!

Consequently, we want to do some smoothing or discounting as well. Essentially, we want to use some algorithm that will give us a non-zero probability for every word.

Laplace Smoothing

Laplace smoothing is to add a fixed number (typically 1) to all counts. All counts that were 0 (impossible by the training set’s count and thus by the N-Gram model) will now have a value of 1, which makes it marginally possible! Other counts will also be increased, which keeps their weights high and not devalued but less proportional.

This method sucks in general because it’s simplistic, but it gives a nice baseline and is practical enough for basic features like text classification, which doesn’t really need to produce smooth output and focuses more on precision (and having every vocab word possible is important).

This has unique effects upon the model. For instance, it will change each individual probability and thus the conditional probabilities, which can change the perplexity of the model to some observable degree even though words that are ungodly rare will still be ungodly rare. This is just a reality of introducing a smoothing method; should you have 100k words in your vocab and even if your training set was so large that many of them have counts in the thousands, you are still adding 100k different counts to the dataset. Maybe not significant when comparing one to the other, but still significant in the grand scheme of things.

Add-k Smoothing

This is just laplace smoothing but instead of 1 you add $k$.

Interpolation

Interpolation is the idea of coputing a new probability by interpolating different models. If we had a 3-gram model and there were no instances of something in the 3-gram, we could reduce it to a 2-gram model and attempt to predict the word that way based on the 2-gram model.

You can further reduce this to a 1-gram model, and so on and so forth. That is, for a trigram probability $P(w_n|w_{n-2}w_{n-1})$, we can compute

The $\lambda$ values are “hyperparameters” that you can tune; however, its important that they sum to 1, so that this remains a weighted average.

There’s a lot of work on interpolation and how to make it better; if you’re truly interested I suggest checking out the EM algorithm, an iterative learning algorithm that converges on the best local $\lambda$ values.

Backoff

In a backoff model, if the n-gram model has no data for a given n-gram, it will back off to the (n-1)-gram model. This is a simple way to interpolate between models, and is a good way to avoid the problem of having a 0 probability for a word. However to keep our probability distribution, we have to scale the probabilities of the higher level N-Grams down in order to save some probability space for the lower level ones.

In practice, it’s easier to use something called stupid backoff, which is the equivalent of putting your hands over your eyes and pretending that the probability distribution rules don’t exist. If you need to back off, you back off. Screw the rules!

Conclusion

Thanks for reading! I hope you learned a lot about N-Gram models and how to evaluate them. This will be the last in a 2-part series about N-Grams; I don’t intend to go into them greatly since this series is also kind of my regurgitation of readings following CMSC848O. Many thanks to Prof. Mohit Iyyer for guiding my learning.

Next articles will focus on ML, in particular neural models since they have advanced to become by far and away the standard in the context of language modeling.

Now I need to stop procrastinating on my cybersecurity and DSA articles.

This will be the first in a brief series on how language models work.

Suppose you have an example sentence, for instance

“The student opened their ____“.

You as a person, what would you put? Well, you might say “book”, or “laptop”, or “notebook”. You, the reader, might have much more ideas than this also. But how do you know these ideas? Where do we get the “next word” from, and how do we complete sentences?

This problem is the entire basis of language models, many of which recently exploding in popularity (eg. ChatGPT, Claude, Perplexity)

Although we very rarely think about it, there must exist some kind of unique way that we as humans understand language that gives us an inference on what would happen next.

Consider this: If this sentence was asked to someone in the 1940s, they would not be able to answer “laptop”. In the modern day, most who read this sentence would answer “laptop”, as they become increasingly ubiquitous and physical textbooks phase out of classroom usage in the vast majority of classrooms.

Thus, our internal “language model”, so to speak, is based on the data we have seen. One intuitive idea, then, is that we as humans quite simply would predict the next word using the most common word that we could think of that fits in that location.

What’s an N-Gram anyway?

Back to our example, we have “The student opened their ____“.

Then, suppose we had every single english sentence ever written! That is, we somehow managed to accumulate the entire corpus of the English language, and found an infinitely powerful and fast machine that could process it. Then, an intuitive idea is that we could simply count the number of times any word appears after “The student opened their ____“.

For instance, suppose that we had the following sentences:

1. The student opened their book
2. Then, the student opened their laptop
3. However, the teacher was displeased to see the student opened their laptop
4. The student opened their notebook
5. However, he then turned to see that the student opened their notebook
6. The student opened their laptop

We can then “Ctrl+F” through our data using some program and find that the word “laptop” appears 3 times, “notebook” appears 2 times, and “book” appears 1 time. Then, since “laptop” is the most common word, we would predict that the next word is “laptop”.

However, this has obvious drawbacks! For one, we would need to have every single sentence ever written in order to predict the next word. Even then, this method fails all the time. Many “brand new sentences” are written every single day, and English is inherently a creative language.

For instance, if I prompt a program with an entire paragraph, it will very frequently NOT find that entire paragraph even if we HAD the entire English language laid out before us and we could somehow store all however many petabytes of data that would be, nevermind process it all in a reasonable amount of time.

So we must settle for the “N-Gram”.

The N-Gram idea is simply to ignore everything but the last N words. For instance, in the above example, we could only consider the 2 last words as a prompt, or “context” for the next word (Note: a prompt differentiates from a context in the sense that a prompt is given by the user while context constantly changes; suppose we need to generate multiple words, then for each new word we add that to the context along with the original prompt). Thus, rather than considering

“The student opened their ____”

We would consider

“opened their ____”

This would require us to store far less data, since we could just precompute every single 2-gram in the English language (given a vocabulary of $V$ words, this would require $O(V^2)$ space, which is still very unreasonable but much better than the infinite amount of data we would need to store otherwise). Note that we compute 2-grams for a 3-gram model, since we need to compute the probability of the last word in a chain of 3 words, given the previous 2 words.

It would also allow us to give responses to much more unique prompts, since we would not have to have seen the entire prompt before to give a response, only the last 2 words.

An example of something like this that you’ve almost certainly seen a lot is your phone autocomplete! You may have noticed that your phone will often try to guess the next word you are going to type, and does a pretty good job on a small scale. If you keep spamming the autocomplete result, however, the phone might give you a super long run on sentence, that makes no sense.

This is because it’s just an N-gram model! (ok this is complicated; a lot of newer phones have started implementing full on tiny LLMs, but this is a story for another article). Thus, it does pretty good only with the context of one or two words, but as soon as you start typing a longer sentence, or something that exceeds the size N of the N-gram, it will start to fail.

Prediction

General Idea

Okay, so we have an idea of how to predict the next word. But how do we actually do it? That is, how do we predict the “most likely” next word?

Once again, a naive method would be to simply count the number of times each word appears after the prompt, and then predict the most common word.

Mathematically, what we’re really asking is what $P(w|h)$ is, where $w$ is the word we want to predict, and $h$ is the history of words we have seen so far. For instance, we might want to compute

$P(\text{computer}|\text{The student opened their})$

We can decompose this sequence using the chain rule of probability as

The Markov Assumption

The Markov Assumption is that context is only dependent on the last few words to some degree. For instance, the assumption that a 2-gram model is sufficient to predict the next word, as opposed to an infinity-gram model.

It should be intuitively clear that this results in a loss of information, and thus accuracy. However, for the sake of computation, we have to let this go for now. Thus, assume for a 2-gram model for instance, that

and in general for some $N$-gram model

Computing $P(w|h)$

The question remains of how we will get these base probability values; we will do so via the Maximum Likelihood Estimation (MLE) method. We get an MLE estimate by counting from the dataset, and then normalizing the counts to get a probability distribution (so that they are in between 0 and 1)

We will find $P(x_n|x_{n-1})$, for instance, by computing

where $C(x_{n-1}x_n)$ is the number of times the bigram $x_{n-1}x_n$ appears in the dataset, and $C(x_{n-1}x)$ is the number of times the unigram $x_{n-1}$ appears in the dataset.

Since the sum of all bigram counts that start with $x_{n-1}$ is the same as the unigram count of $x_{n-1}$, this can be simplified to

To elaborate; the sum of all bigram counts is the same as the unigram count because we’re counting the number of times any word appears after another word, and then summing this; but this is the same as just counting the number of times the first word appears.

For the general case of an n-gram model, we can then compute

Note that this estimates n-gram probability by dividing a sequence frequency by its prefix frequency; we call this relative frequency.

Scale

In practice, language models are huge. Thus, we may be multiplying together numbers between 0 and 1 many, many times. If you took 5th grade maths you may recall that multiply two numbers between 0 and 1 results in an even smaller number; this will eventually result in data underflow, which would be catastrophic. To prevent this we frequently use log probability as the method of storage.

Recall from your high school maths that $\log(ab)=\log(a)+\log(b)$; thus, we can convert our multiplication into addition, which is much more stable. In general, where $p_i$ is some probability,

Next

We’ll discuss a broad overview of training and evaluation, along with entropy next, to continue the N-Gram series. We will eventually advance to neural models and modern attention, transformer, and scaling ideas.

I hope to continue this series alongside working on the ADS series and covering other mid level ML concepts.

The US FDA, or Food and Drug Administration, has this huge responsibility which is approving medical devices for use in the United States. The 510K database is a collection of all the devices that have been approved for use in the US. Maintained online, it’s extremely primitive and is, honestly, horrible to work with as a developer. There is no API, and the only real way to do anything is web scraping (which I hate to do on a government website, so ideally we will want to keep this to a minimum).

What is a 510K?

A 510k is the submission form made to the FDA to get a medical device approved for use in the US. It’s a long, detailed form that includes a lot of information about the device, its intended use, and the testing that was done to prove that it’s safe and effective. More importantly, this information can include the intended use, restrictions, etc.

Now say that we want to search for all devices under a certain category with a keyword in the 510k. How do we do that? uhh… turns out there isn’t really a native solution, and because this is such a niche problem, there isn’t really a third party solution either.

It turns out that the FDA has this database, but has literally nothing to search it with. About 4 months ago, I wrote a short (very inefficient) script to do it myself. It’s written in python and is found here.

This isn’t a medical device/regulations site, so lets go into the code, which is the actually interesting part.

The Script

The idea is essentially to scrape the site using regular URL patterns. I used BS4 for this, but because of how incredibly simple the scrape is you can probably get away with writing your own protocol or using a different library.

Imports:

python
# data related libs
import pandas as pd
import csv 
import pickle
# ocr
from tempfile import TemporaryDirectory
import pytesseract
import pdf2image
from PIL import Image
import requests
import cv2
# web scraping w/ bs4
import httplib2
from bs4 import BeautifulSoup, SoupStrainer

I’m using pandas because we’re actually going to be working with a considerable amount of data here. The CSV library is used because I’m going to be loading the initial code files from a CSV that the FDA helpfully provides here. The pickle library is used to save the data in a way that can be loaded quickly. The script takes a while to run because we’re going to OCR a lot of PDFs, so we want to be able to stop and restart the code without having to re-do everything.

Global Variables

py
# -------------------------
# GLOBALS
# -------------------------
# 1: read from csvs, generate lists, save them
# 2: read from lists, scan pdfs, create txts
# 3: scan through txts for keywords
start_idx = 0
mode = 2
# file prefix
DATADIR = "data/"
# data names (eg. the names of the csvs)
DATA = ['96cur','7680','8185','8690','9195']
# pdf prefix
PDFDIR = "pdf/"
# csv delimiter
DELIM = '|'
# valid product codes
VALID_CODE = ['GEI','PAY','ONQ','OHV','GEX','OHS','NUV','ONG','ONE','ONF']
# keywords - all lowercase
COND_KEYWORD = ['wrinkle']
KEYWORD = ['fitzpatrick','scale','type']
# DB Link (510k)
DBPREFIX = 'https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID='

I end up writing this script to have 3 modes.

• The first one is a first pass, which just creates a list of the possible 510k files of interest. It will sort out the ones that don’t have a public summary file (eg there’s almost no details publicly available, so isn’t very useful).
• The second one goes through the lists of the 510k files and scans them for keywords. It will OCR the PDFs and save the text files in the data directory.
• The third one goes through the text files and scans them for the keywords. I ended up not implementing this because it’s actually slightly redundant given the second step.

In my above example code I also have some basic file/directory names, the delimiter for the CSV that the FDA publishes (which is |). Then I speciy my search conditions. Here we will look for any one of the keywords if and only if the statement has the conditional keyword, and we will search all files under those product codes.

Helper Functions

py
# -------------------------
# DATA/FILE SAVING/LOADING
# -------------------------
# Load codes from csv    
def load_csv(input):
    data = pd.read_csv(input, sep=DELIM, header=0, keep_default_na=False)
    return data # returns df
# load a list from a text file
def load_txt(input):
    lst = list()
    with open(input,'r') as f:
        for line in f:
            lst.append(line.rstrip())
    return lst
# Load a list from file
def load_obj(file):
    with open(file,"rb") as f:
        obj = pickle.load(f)
    return obj
# output a list as a text
def write_txt(out, lst):
    with open(out, "w") as f:
        for item in lst:
            f.write(f"{item}\n")
# write an object to file 
def write_obj(out, obj):
    with open(out, "wb") as f:
        pickle.dump(obj,f)
# read csv files
def read_single(file):
    return load_csv(file)
# read multiple csv
def read_multiple(arr):
    return pd.concat(map(lambda date: load_csv(f'{DATADIR}{date}.csv'), arr))
# -------------------------
# FILTERING
# -------------------------
# return the rows in a df that match a value in the arr
def filter_by_col_arr(pd_arr, column, value_arr):
    return pd_arr.loc[pd_arr[column].isin(value_arr)]
# return the rows in a df that match the value
def filter_by_col(pd_arr, column, value):
    return pd_arr.loc[pd_arr[column] == value]
# returns the column as a list
def get_col_as_list(pd_arr, column):
    return list(pd_arr[column])

An assortment of helper functions that I wrote for a few different formats so I can call them quickly in the code later

Scraping

py
# gets the link for a summary or statement from the FDA db by scraping
def getlink(type, prefix, code):
    http = httplib2.Http()
    status, resp = http.request(f'{prefix}{code}')
    for link in BeautifulSoup(resp, features='html.parser', parse_only=SoupStrainer('a')):
        if link.has_attr('href') and link.string==type:
            return link['href']
    return ""

This is an incredibly simple bs4 function that I wrote to get the summary argument from the FDA database. It simply takes a link and looks for the first link that has the text “type” in it. We pass in Summary later, so it looks for the link with the text “Summary” in it.

The URL scheme of the 510k DB is the most important, and is a little bit tricky to figure out because it’s not very well documented. The URL for the codes we are interested in are in the form of https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID={code}. The code is the product code, and the FDA has a list of valid product codes that they use.

However other than CDRH’s 510ks, there are other departments, sometimes there are weird prefixes for the codes, etc. Specifically in the context of this search, these parameters work. If you want to adapt this script for other purposes you definitely need to change this part.

Main Function

py
# -------------------------
# READ/OCR PDF
# -------------------------
# Reading the text. There are multiple cases to this.
# Case 1: OCR correctly reads text. In this case, we concatenate all text and 
#       save it as a .txt file in PDFDIR/ocr/{code}.txt. Also return "success" or 1
# Case 2: We can't read it. Then, we return "fail" or 0. This will get concatenated to the 
#       "none_knums" variable and outputted as a text file.
def pdfscanner(type, prefix, code):
    # vars
    img_lst = []
    # open the file online and then create a pdfreader instance
    url = getlink(type, prefix, code)
    if url == "": 
        return 0
    # get the actual content.
    pdf = requests.get(url, stream=True).content
    # We use OCR to recognize the text. 
    # We can use PdfReader to find the DPI.
    # reader = PdfReader(bytes_stream)
    print(f"{type} of {code}: URL {url} | DB URL: {prefix}{code}")
    
    # Implementation of case 1:
    with TemporaryDirectory() as tempdir:
        # Step 1, turn the pdf into images.
        # read the pdf as images
        pdf_pages = pdf2image.convert_from_bytes(pdf)
        for num, pg in enumerate(pdf_pages, start=1):
            fname = f'{tempdir}/{code}_{num:03}.png'
            pg.save(fname,"PNG")
            img_lst.append(fname)
        # Step 2, read the images
        # open the txt file output
        with open(f'{PDFDIR}/ocr/{code}.txt','w') as f: 
            for img in img_lst:
                # OCR the page
                # image preprocessing can be put under here
                image = cv2.imread(f'{img}')
                page_txt = str(pytesseract.image_to_string(image, lang='eng', config='--psm 6')).replace("-\n","")
                f.write(f'{page_txt}\n')
    return 1

This is the bulk of the work done. You can tell that there’s quite a few phases here.

First, some error handling, and attempt to ""save"" the PDF. If it fails we leave and try the next one, noting it down in an output file. Otherwise, we move onto the OCR step. I used pdf2image to convert the PDF to images, and then pytesseract to read the images. I save the text to a file in the PDFDIR/ocr directory.

Driver Code

py
# -------------------------
# DRIVER CODE
# -------------------------
if mode==1:
    # read the csvs in as df
    csv = read_multiple(DATA)
    # find results by product code
    results = filter_by_col_arr(csv, 'PRODUCTCODE', VALID_CODE)
    
    # find results with summary
    results_summary = filter_by_col(results, 'STATEORSUMM', 'Summary')
    results_statement = filter_by_col(results, 'STATEORSUMM', 'Statement')
    results_none = filter_by_col(results, 'STATEORSUMM', '')
    # get the knumbers
    results_knums = get_col_as_list(results_summary, 'KNUMBER')
    summary_knums = get_col_as_list(results_summary, 'KNUMBER')
    statement_knums = get_col_as_list(results_statement, 'KNUMBER')
    none_knums = get_col_as_list(results_none, 'KNUMBER')
    # write to files
    write_obj(f'{DATADIR}matching_codes_pickle',results)
    write_obj(f'{DATADIR}matching_codes_with_summary_pickle', results_summary)
    write_obj(f'{DATADIR}matching_codes_with_statement_pickle', results_statement)
    write_obj(f'{DATADIR}matching_codes_none_pickle', results_none)
    write_txt(f'{DATADIR}matching_codes.txt', results_knums)
    write_txt(f'{DATADIR}matching_codes_with_summary.txt', summary_knums)
    write_txt(f'{DATADIR}matching_codes_with_statement.txt', statement_knums)
    write_txt(f'{DATADIR}matching_codes_none.txt', none_knums)
    # print some numbers
    print(f'Total files: {len(results)}\n'
        f'Matching files with a knumber and summary: {len(summary_knums)}\n'
        f'Matching files by summary: {len(results_summary)}+{len(results_statement)}+{len(results_none)}')
elif mode==2:
    summary_knums = load_txt(f'{DATADIR}matching_codes_with_summary.txt')
    print(f'Successfully loaded {len(summary_knums)} nums!')
    # vars
    failed = []
    success = []
    # create the txt files for each knum with a summary
    for num, knum in enumerate(summary_knums[start_idx:]):
        if pdfscanner('Summary', DBPREFIX, knum) == 1:
            print(f'Successfully converted #{num+start_idx}: {knum} to txt')
            success.append(knum)
        else:
            print(f'Failed to convert #{num+start_idx}: {knum} to txt')
            failed.append(knum)
    write_txt(f'{DATADIR}converted_to_txt.txt', success)
    write_txt(f'{DATADIR}failed_to_txt.txt', failed)
elif mode==3: 
    print("Implement mode 3")
else:
    print("How did you get here? Wrong mode #.")

This stuff is pretty simple and mostly involves a lot of logging and file saving/writing etc. You can see where I call OCR in mode 2, and note that we are looking for the “Summary”. There are other types of documents that the FDA has, but I only cared about the summaries. Note mode 3 not implemented because we already have search by keyword anyway. It wouldn’t be too hard to write though, and I have other FAR scripts on my github that do this.

Summary

This script is a bit of a mess, but it works. It’s not the most efficient thing in the world, but it’s a good example of how to scrape a website that doesn’t have an API. The FDA’s 510k database is a good example of this, and I hope that this script can be useful to someone. I’m not going to be maintaining it, and I certainly will not do something like this again, as I only did this for the sake of finding this very specific query. Anyway, I hope this was helpful to someone.

• TOC
• The Problem
• Technical Details
• Approach
  • Initial Thoughts
  • How to Fit?
  • The DFS function
• Solution
• Thoughts

The Problem

I present to you today the Polyomino problem from an AtCoder contest I did. This was the prompt provided

The problem immediately struck me as being relatively complex, and I wanted to walk through the interesting approaches made to solve it. For those interested in the final solution, you may want to consider skipping ahead.

Technical Details

This was completed as part of an atcoder competition, so I’ll be working with C++ and typical competitive programming rules here. I specifically wrote this code under some time duress, so apologies if it’s a little bit scuffed.

Approach

As you may have noticed, the problem is immediately very complex. We receive 3 4 by 4 “input images” of our polyominos, and we first need to be able to read those. That was relatively simple, and I decided to go with a vector array of strings to scan these in due to how manipulatable they are.

Then, we must determine whether or not they can be put in a grid together.

Initial Thoughts

It’s immediately obvious that there are some conditions where we instantly fail. For instance, if there are not enough #’s inputted to actually fill a square, then we can instafail. Additionally, since the polyominos can be manipulated under translation and rotation, we need a rotation algorithm. Let’s get some basics out of the way first.

cpp
// INPUT
vector<array<string, 4>> g(3);
for (auto &it: g) {
    for (auto &i: it) {
        cin >> i;
        cnt += count(i.begin(), i.end(), '#');
    }
}

The input method is extremely basic. We just define our vector array of strings, and read them in. We also track how many #’s are inputted so we can check if there are 16 of them, as mentioned earlier. I used auto &i : it, the easy range for conditional, to make things easier. You could of course do this with a more traditional iterator.

cpp
auto rotate = [](const array<string, 4> &a) {
    auto b = a;
    for (long long i = 0; i < 4; i++) {
        for (long long j = 0; j < 4; j++) {
            b[j][3 - i] = a[i][j];
        }
    }
    return b;
};

And this is the rotate function. Notice that I just defined it within main. The algorithm itself is very simple; we create a new array b, which is initially uninitialized, and then transpose characters 1 at a time at a 90 degree rotation into b. Since we know the size of the array, we can just iterate over a for loop. In retrospect using an ll iterator was a waste of memory but it’s just a product of the int->ll macro I had.

How to Fit?

The question of how to fit the polyominos into the actual square was the harder part. My initial naive approach was to try every single possible rotation and location. In actuality, there is no “logical” way to determine how a puzzle piece fits into a puzzle. In the classical jigsaw puzzle question, and assuming there isn’t information on what color something is, for instance, our solutions typically involve either the naive approach of trying everything or to create an adjacency matrix of what fits against what and then pruning it.

However, in this problem an adjacency matrix doesn’t make much sense. We don’t care so much about how pieces fit together with each other, as whether or not they’re all able to, within a 4x4 array. In this case, each piece has 4 rotations and 16 theoretical locations at most. That is, the worst case scenario checks 4^16^3 possibilities. This is insane, and we don’t want to try this.

Let’s think of a way to reduce this.

We can treat the problem like a search and create a dfs algorithm. This algorithm will look at each possible rotation, and on each one attempt to fit them into the block in a way that satisfies the solution. However, to minimize the amount of checks we make, we want to retain “memory” of what is actually possible.

To do this, I defined a 2d array “page” where we put the imaginary blocks. The page will be a reference point of where a “valid” placement of a block has already been found.

Writing a fit algorithm is actually relatively easy. we can just check all 16 locations and if the page is blank and the input polyomino is there, we can slot it in. Making sure to check for edges and other edge checks, we get this

cpp
auto fit = [&](long long k, long long x, long long y) {
    for (long long i = 0, x_iter = x; i < 4; i++, x_iter++)
        for (long long j = 0, y_iter = y; j < 4; j++, y_iter++) {
            if (g[k][i][j] == '.') {
                continue;
            }
            if (page[x_iter][y_iter] == '#') {
                return 0;
            }
            if (x_iter < 0 or x_iter > 3 or y_iter < 0 or y_iter > 3) {
                return 0;
            }                
            page[x_iter][y_iter] = '#';
        }
    return 1;
};

It’s a pretty simple solution.

The DFS function

There are plenty of different possibilities however, so we have to write the dfs function in a way that allows us to look at various possible fits and rotations. I wrote it in a recursive manner, where the base case is either failing after all rotations have been tried or when a spot has been found. I used a tmp variable to temporarily “save” the page, and if it’s valid I would “replace” it.

cpp
auto dfs = [&](auto &&self, long long i) -> bool {
    if (i == 3) {
        return 1;
    }
    auto tmp = page;
    for (long long x = -3; x <= 7; x++) {
        for (long long y = -3; y <= 7; y++) {
            for (long long t = 0; t < 4; t++) {
                if (fit(i, x, y) and self(self, i + 1)) {
                    return 1;
                }
                g[i] = rotate(g[i]); 
                page = tmp;
            }
        }
    }
    return 0;
};

Using tmp to save the page allows me to only take pages that are successful. We try to fit the shape in from -3 to 7 because the polyomino we receive as input isn’t always started at input[0][0], so to speak. It could be at the input’s input[2][3], at which point given how the fit algorithm is written we’d have to start at -2, -3. Since it can only be at most 3 to either side, we go from the range (0, 4) to (-3, 7).

Then, we try all 4 rotations of the shape and check if it fits. If it doesn’t, we rotate it and try again. Notice that our input structure g only has 3 elements (since we took in 3 polyominos) so we have to halt the recursion after all 3 have been rotated 4 times. These 3 functions are actually sufficient to solve the problem!

Solution

Now, let’s put it all together. I used a simple check at the end to determine whether or not the polyominos would fit, at which point I would print YES or NO correspondingly.

cpp
#include <bits/stdc++.h>
using namespace std;
int main() {
    vector<array<string, 4>> g(3);
    array<array<char, 4>, 4> page;
    // unrelated note but count is actually a function
    // name in stdc. mildly infuriating.
    long long cnt = 0;
    // taking input to the problem
    for (auto &it: g) {
        for (auto &i: it) {
            cin >> i;
            cnt += count(i.begin(), i.end(), '#');
        }
    }
    // rotate function
    auto rotate = [](const array<string, 4> &a) {
        auto b = a;
        for (long long i = 0; i < 4; i++) {
            for (long long j = 0; j < 4; j++) {
                b[j][3 - i] = a[i][j];
            }
        }
        return b;
    };
    // fit. note that page is global here so it can be used by
    // both fit and dfs. this is important because
    // we want it to "remember" what works.
    auto fit = [&](long long k, long long x, long long y) {
        for (long long i = 0, x_iter = x; i < 4; i++, x_iter++)
            for (long long j = 0, y_iter = y; j < 4; j++, y_iter++) {
                if (g[k][i][j] == '.') {
                    continue;
                }
                if (page[x_iter][y_iter] == '#') {
                    return 0;
                }
                if (x_iter < 0 or x_iter > 3 or y_iter < 0 or y_iter > 3) {
                    return 0;
                }                
                page[x_iter][y_iter] = '#';
            }
        return 1;
    };
    // This is the search. We stop at i=3, which is kind of
    // the "accumulator"-esque part of the recursion.
    auto dfs = [&](auto &&self, long long i) -> bool {
        if (i == 3) {
            return 1;
        }
        auto tmp = page;
        for (long long x = -3; x <= 7; x++) {
            for (long long y = -3; y <= 7; y++) {
                for (long long t = 0; t < 4; t++) {
                    if (fit(i, x, y) and self(self, i + 1)) {
                        return 1;
                    }
                    g[i] = rotate(g[i]); 
                    page = tmp;
                }
            }
        }
        return 0;
    };
    // If the count isn't 16, we can auto fail.
    // Only then do we call dfs. 
    // Note if cnt isn't 16 then dfs never runs. 
    if (cnt == 16 and dfs(dfs, 0)) 
        printf("Yes\n");
    else
        printf("No\n");
    return 0;
}

Thoughts

The problem was really interesting to solve during the time pressure of a competition. In retrospect, the code could have been a lot cleaner. For instance, I could have redesigned the dfs code to not be so for loop intensive, since it deterministically goes through 400 recursions every layer.

Additionally, I think that you can tell the code was very clearly written for competitive programming. After all, there’s so much auto used.

I think that the problem demonstrates how “geometric” problems that we typically use human intuition for (should you approach a problem like this in real life, we tend to just “intuit” possible block placements, and don’t really think about why our brains can conduct such pattern recognition so efficiently) can be solved more easily through compartmentalization. This problem was relatively simple, with many constraints and a small “puzzle” size. Yet it still required multiple compartmentalizations and solutions for each of them.

Intro

I am not great at translating or at Japanese or Chinese specifically. I do, however, have some layman’s understanding of both, and my family background helps a lot with that (since I’m Chinese). So when I noticed that a certain Yorushika song wasn’t translated, I hopped on the chance to get some practice in and translate it.

As it turns out, no one had ever translated it before, and the video has become somewhat popular, getting 820k views at the time of writing. It has buoyed me into trying other translations, and I’ve made 2 more, one for an advertisement (yes, an advertisement) whose animation I found particularly compelling, and a nao song. They’ve been incredibly interesting, and I wanted to talk about the benefits of doing song translation and how I did it.

What I think

It turns out that learning language without practice is incredibly difficult. And as it turns out, living in a completely different country (the US) to either language you’re trying to become more familiar with is not helpful at all. It helps greatly, therefore, to try to translate something that has a wide range of vocabulary as practice. This helps with keeping things fresh in your mind, and frequently in learning new words, since songs and books have all sorts of crazy vocabulary that you’d never see anywhere else.

It also helps that if you’re translating something, you probably have a lot of interest in it, and thus it’s far more engaging than reading a textbook or grinding Duolingo lessons/online quizzes.

So translating music has been incredibly fun for me. In the working of the nao song, I spent many hours trying to think of the best equivalent english to translate it to, and how to best communicate the emotion that nao wanted to get across. An accurate translation doesn’t just have direct accurate vocab translation, but also brings across nuance. This is particularly hard to do in a song with little understanding of slang, so I spent a lot of time googling sayings in order to get an accurate idea of what nao was trying to represent. In fact, I was actually kind of surprised by my final translation, since it was actually slightly different than my prior impression of the song while just casually listening to it (of course, I maybe only truly understood half the song before).

I started translating the song 3 hours before one of my college classes… I accidentally ended up skipping it with how engrossed I was with the TL. Fortunately, the class wasn’t particularly important, although I’ll never admit that to the professor in question. But I did find it incredibly gratifying and a great learning experience.

I strongly encourage any of you reading this who are trying to learn a new language to try translating a song or novel. They push you to understand quirky words (if you just look at textbooks, they’ll only try to teach you the standard language) and help you understand slang and nuances you’d never see anywhere else. It’s also fun! That’s not to say that you should ever replace standard models of learning like the classroom or textbooks with soley hobby translation like I am. I am not remotely serious about learning Japanese or Chinese to their fullest, and I don’t plan on moving there soon. But it’s a great way to practice.

Windows is not UNIX. This is a fact that we cannot, unfortunately, change. For the longest time, I used cygwin to ""make do"" when writing C code - at least it could compile, right. But Mintty and the emulation that cygwin provided always missed a little bit of something, especially since it was just a compatibility layer for POSIX and not the real thing.

Windows came out with WSL ages ago for me. I’m not very old, so it came out when I was in middle school, and I didn’t even clearly know what the difference between Matlab and Java were at the time, so it didn’t really matter for me. But I did hear from many people that WSL sucked compared to cygwin or MinGW, so when I started coding in late middle school into high school, I too installed cygwin. Cygwin is incredibly easy to use, you install it, the files are still easy to find in your C: drive, and you could legitimately edit your .c files with notepad and then just compile them on cygwin’s cli if you wanted.

WSL 2 came out a few years ago, and I started using it this summer and I cannot express how nice it is to have an actual kernel. It’s true that it is ""less convenient"" than cygwin or mingw still in the sense that the files are less accessible and you need to go through the trouble of starting WSL up, but for people who aren’t interested in using linux as an actual operating system but still want as complete of a UNIX based system as possible for the sake of programming, it’s pretty amazing. It works with everything (with minor differences)!

I frequently ssh’d into some linux server for the sake of running programs in a more complete or powerful environment (typically because they had money and could have a lot of GPUs) and while WSL is not better than this in the slightest, it does give that feeling off. It’s certainly not fair to compare WSL to cygwin since WSL is basically just coLinux while cygwin is just a compatibility layer, but I do think that WSL is easier to use and install than cygwin or mingw. And I do think that WSL is easier for new programmers to approach than cygwin is.

All told, I think this is a rare case where Windows actually pulled something off for the sake of programming that I wholeheartedly enjoy using. Good job. But I wrote this for one thing, which is to encourage any readers to at least try cygwin. It’s a cool experience and incredibly useful for learning a lot of basics, and I would hate for it to be so overshadowed by WSL (although deservingly so) that the experience and knowledge you get from using it is completely lost to history.

Intro

So I play this game called osu!, which is pretty cool and could probably warrant a post in my personal section by itself. Regardless, I basically play a really rare (and objectively pointless, but subjectively cool) mod called Spunout, which makes the game basically no different than before, aside from removing an object called the spinner. The gameplay details of the mod can be left out, however - the points I need to make are first, that players in osu! sometimes ""main"" a mod (in which they play the majority of the game with that mod), and additionally that Spunout is incredibly rare.

I used to be part of a discord server with 5 or 6 people in it where we all played Spunout. It was created purely from just knowing people who you knew played the game. But it died… years ago. I recently met a player called renyoo while reffing a tournament, and I had seen them around the mod leaderboards lately, so I struck up a conversation, and found out that they were really interested in the idea of a spunout server. So we made one, except I wanted to get everyone this time, and not just the few people who we knew. So we wrote a script! Also, for those interested, here is the csv file I used for the countries, since I couldn’t find a complete list of the osu! countries online that was accurate

Technical stuff

More specifically, I wrote the first version of the script. Fortunately, I didn’t have to do any web scraping since the developer of osu!, peppy, is very nice to developers and provides an extensive API with an incredibly generous limit (2k requests per minute, at the time of writing, although I hear this will be reduced in the future). So I just did some basic query calls and came up with something that would page the top 10000 players from every country leaderboard, which altogether gave me around 800 thousand user IDs. I wasn’t really keen on learning a new language for the sake of this project and I was pretty lazy, so I just wrote it in python:

python
import ossapi
from ossapi import Ossapi
import time
from tqdm import tqdm
import pickle
import pandas
import multiprocessing
from concurrent.futures import ThreadPoolExecutor, as_completed
# I created a client using my id and secret here but you don't get to see it ;)
# create api
api = Ossapi(client_id, client_secret)
#... some code
def get_users_country(country, scorelist, out):
    print(f"Start: {country}")
    user_ranking = api.ranking("osu", ossapi.RankingType.PERFORMANCE, country=country)
    api_count = 1
    curr = user_ranking.cursor
    user_list = list()
    # while curr checks to ensure that there remains pages
    while curr and len(user_list) < 10000:
        for j in range(0,len(user_ranking.ranking)):
            # add user to list if they are neither in the score ranking nor pp ranking
            # global rank can actually be undefined in some cases
            if (not user_ranking.ranking[j].is_ranked) or (int(user_ranking.ranking[j].play_count or 0) < 2000) or (int(user_ranking.ranking[j].global_rank or 0) > 200000):
                #print(f"Skipping user since they're too low")
                break
            elif int(user_ranking.ranking[j].global_rank or 0) <= 10000:
                #print(f"Skipping user since they're top 10k")
                break
            elif user_ranking.ranking[j].user.id in scorelist:
                #print(f"Skipping {user_ranking.ranking[j].user.id} as they are in score index")
                break
            else:
                user_list.append(user_ranking.ranking[j].user.id)
        user_ranking = api.ranking("osu", ossapi.RankingType.PERFORMANCE, cursor=curr, country=country)
        api_count += 1
        print(f"{country} Progress: {len(user_list)}/10000, with {api_count} calls")
        curr = user_ranking.cursor
        # api cooldown
        # time.sleep(.5)
    if not curr: 
        print("Ended due to curr")
    with open(out,"wb") as f:
        pickle.dump(user_list,f)
    print(len(user_list))
    return user_list
#... more code

A lot of this project ran into early errors when I forgot to account for silly mistakes like making sure that there were actually 10 thousand players in any one country’s leaderboard (regions/states with little internet presence like American Samoa for instance would not have 10k), or checking if they were inactive (osu! would change their profile statistics to effectively “delist” them from the rankings, but the api call caught them and attempted to process them anyway)

I included pickle because python ended up being a super slow language, so I decided to split up the “getting users” (which would use relatively few api calls and calculations since I could get 50 players at a time) and actually checking for Spunout players (which would have been much more calculation and api intensive, with at least 2 calls per player). So I had dumb little helper functions like

python
# Load a list from file
def load_users(file):
    with open(str(file),"rb") as f:
        user_list = pickle.load(f)
    return user_list
# Load country codes from csv    
def load_csv(input):
    data = pandas.read_csv(input, header=0)
    return list(data.Code)
# Load users from a txt document (The easiest way to save user lists between scripts of different languages was .txt)
def load_txt(input):
    user_list = list()
    with open(input,'r') as f:
        for line in f:
            user_list.append(line.rstrip())
    return user_list

Regardless, this finished the task of getting the hundreds of thousands of players reasonably quickly, taking me about an hour (intentionally slowed down to rate limit myself - i wanted to stay at 60 req/min even though I had 2k). Then, it was just the task of checking every single individual player

Well, I wrote a short script to check players too, before realizing that the script would take (about) 140 hours to finish, even running about 1k req/min. That wouldn’t do. I started thinking about rewriting the whole thing in Java, before one of the people that we had invited already, Kuwume, simply saved me by writing it in C++. It’s significantly faster, and gets the job done much easier. I’m not going to link the script, but it did the job in about 6 hours as opposed to 140. He also capped out the API limit instead of taking it slow like I did.

The Sheet

I wanted to create a google sheet where you could see all the spunout players and their stats and how they rank against each other, and it’s finished, and at this link. But this one was also an adventure too! To detail, I did this using an apps script, which uses the list of Spunout players by ID that Kuwume and I generated in the previous bits. However, using Google Sheets’ App Script to update the sheet (which, in hindsight, might have been easier if I had simply just hosted the leaderboard on this website) was really troublesome due to the Sheets rate limits. I couldn’t call the API many times per minute, so I had to add delay into the script. I couldn’t update the sheet a certain amount of times per day, so I set it to only update every 6 hours. I couldn’t expose my API key, while Apps Scripts are public to literally everyone who can make a copy of the sheet, so I had to create a private sheet and a public sheet where I would do all the processing and filtering and raw data manipulation on the private sheet, and then use the IMPORTRANGE() function to get them onto the public sheet.

Since I didn’t want to write JS, I ended up asking someone who had created a similar script if I could have the code for how he was calling the API from Apps Script, and then modified that code accordingly for my needs.

But that created a problem, since the IMPORTRANGE() function doesn’t literally import the cells, but rather makes a shallow copy of the data presented to the viewer. Thus, I couldn’t import any functions (or images!), so I ended up writing the script in a really funny way to split up things and then using ARRAYFORMULA() nonsense to regenerate the images/etc. This was an adventure for me, where I learned all about conditional formatting, all the verious quirks of google sheet formulas, etc. I also had to write in a lot of forumula checks to make sure that I was actually generating on a person to prevent errors too! I ended up designating a “reference cell” range where I’d calculate some numbers that were important to formulas, such as how many players I had, and then reference those later. But this ended up in some funny formulas, like =QUERY(IMPORTRANGE("1cmqKgZdvNmdPWVgm5HYK4m5jjXe4W1fp_8DYLTNsGH4","Raw Data!B1:O"&IMPORTRANGE("1cmqKgZdvNmdPWVgm5HYK4m5jjXe4W1fp_8DYLTNsGH4","Raw Data!P21")),"select * order by Col4 ASC") where you can see I was using the reference cell for the number of players.

Regardless, I think the end sheet looks pretty good! And here is the apps script snippet responsible for getting people:

js
// takes an array of user ids and returns the corresponding user objects from the API
function getUsers(userids, token) {
  let headers = {
    'Authorization': `${token.token_type} ${token.access_token}`
  };
  let reqs = userids.map(user => ({
    'url': `${ENDPOINT}/users/${user}/osu?key=id`,
    'method': 'get',
    'headers': headers,
    'muteHttpExceptions': true,
  }));
  let responses = UrlFetchApp.fetchAll(reqs);
  let userObjects = responses
                    .map(e => JSON.parse(e.getContentText()))
                    .map(e => {
                      if (e.error !== undefined) {
                        let user = Object.assign({}, defaultUser);
                        return user;
                      } else {
                        return e;
                      }
                    });
  return userObjects;
}

with similar code used to check the top 100 plays of each user for spunout occurrence.

One of my uni classes last Fall was an introduction to Computer Systems, and during the last month we were introduced to various shells like bash and tcsh. Well, during the class we had also learned about everything required to make a shell, so that’s awfully convenient isn’t it?

Thus, I have recreated (to a very limited degree!) some elements of the bash shell.

It’s worth noting that I’m actually running this command via SSH, so when I input eg cd, it actually gives me the dir of my Windows VSC workspace.

I named the shell “badsh”.

About the Code

Given that this code heavily involves work from projects that I wrote in class, I can’t release the code (since otherwise the University people will give me a big no no and I’ll probably cry myself to sleep after getting kicked out of the school). However, there are some interesting things I’d like to share.

The first, this was written in C. We are working with linux/bash here, so it was a given. We used a prebuilt bison parser during our class in order to parse cli input, but I decided that I wanted to be cool and wrote my own. Consequently, the code runs inefficiently and has many issues with parsing some specific patterns (eg giving it inputs for a manually implemented command (eg the whoami command, which simply returns the code author) with arguments that are nonascii characters will break it).

The general algorithm is structured like so:

• Read input
  • If the buffer is 0, break
  • Otherwise, read the buffer into command history (so we have command recall functionality)
  • Parse the input
    • This works by attempting to first find certain flags, such as | or & or > or < (input redirection). Then,
    • Construct a tree and recursively call the parser left or right until you finish building the tree(Some flags have more priority than others to keep in line with bash, of course)
  • Call executor sub program
  • Repeat until the heat death of the universe or until we exit

The executor program is relatively simple,

• Accept input in the form of our parsed tree
  • Starting at the top of the tree, make our way down, recursively calling the executor on the left and right side.
    • If left fails, kill everything
    • Depending on which flag we’re seeing on our tree, we may want to start a pipe or read/write to a file. Implementations for those are there in the form of various forks and pipes via dup2.
    • If we see a pipe or subshell, go through children in order to isolate the environment
  • Check input against manually implemented commands, eg “exit” or “cd”, which concern themselves with the special environment rather than linux’s native commands, or fun manually implemented ones like “whoami”
  • Call an exec function in order to use linux’s native commands

Seems pretty simple, yea? Unfortunately, this took me the better part of my winter break. I started as soon as I got off school 2022-12-20 and now its 2023-01-05 and I’m finally showing this off.

Hello! I got a lot of help from about entry requirements and skimming places to go on various online forums, so I figured I’d share my experiences and maybe help allay some travel concerns for anyone planning to go to Korea late summer. I went with my father and younger sister, both of us Asian but not Korean (with zero Korean speaking skills - this was hilarious)… we also booked super annoying flights (not so hilarious)

We are US Citizens, routing through Canada, but with no landside stop. This would require you to take a Covid test in Canada, because it resets the clock, so if you’re doing a connection and don’t have the time to wait for your test results to come back, be careful!

Other notes: 1.) we came during the supposed height of monsoon season, but we only got rained on a single afternoon despite every “next day’s” forecast being rain. Lucky. 2.) ALWAYS CARRY YOUR PASSPORTS! They are your only form of legal ID and are super useful. 3.) This trip was kind of supposed to mix a bit of “daily life in Korea” into the mix, so we looked at a lot of not-typical things, like huge HomePlus stores, the Costcos there, the local markets. My dad’s friend showed us around Suwon, his apartment, the workplace, and his son showed us around the schools, recreation, etc. It’s very cool.

Entry

July - (Before Trip): Filled out K-ETA. We’re American. Got approved within 20 minutes. Made sure to specify hotel address, and email is important!! Didn’t have a Korean number immediately so filled in my US one, but you will want to have an email for all your entry things to keep them together before you pick up a sim card. Downloaded Naver Maps (in english mode) and Kakao T, and made an account for Kakao T. Google maps is inadequate (it draws walking lines through buildings, has weird transit directions sometimes, and has completely wrong operating hours for most businesses/places) and Kakao T is horribly important when you leave a major city. We packed somewhat light so we didn’t need the taxis to get to and from airports.

July 7th: Took a PCR test at the local CVS at 14:00 or so. They satisfied the PCR test requirement because it was 2 days in advance of our first flight to Canada, and even though it was over 48 hours, it satisfied the “48 hours until 0 hour” rule Korea has. Most of you won’t need to bother with double rules like this, because you don’t have a 40 hour flight plan.

July 8th: Received PCR tests from CVS at night. Tests said they’d come back in 1-2 days so this was nice because we had more time to fill out Q-Code. I filled out the Q-Code forms for all three of us and they returned the QR Codes to me in an email within an hour.

July 9th: Checked into BWI. Flew to Canada. Found out even if you don’t enter landside, you still must do ArriveCAN. IF you’re flying a Canadian connection and don’t plan to enter the country, do it beforehand. After landing in Toronto, stayed airside through International Connections (fairly fast process), got settled down to sleep.

July 10th: Got onto the Seoul flight, painful delays.

July 11th: Landing in Seoul/Incheon! We were given forms on the plane for Covid declaration (but only if you hadn’t done Q-Code), customs declaration, and a visit card. We filled out the latter two. After landing, they ushered us towards lines where people checked our Q-Code forms (super smooth and fast), and then sent through customs, immigration, and picked up our bags.

We did our Covid tests at the airport outside, at the Transportation Center. I had read on this subreddit that the free public health centers were starting to check for long term residency for foreigners, so we played it safe and paid for tests. We booked ahead through the GoSafe website and set the appointment 1.5 hours after our flight was meant to arrive to be ultra safe, but we arrived slightly late because of flight delays (thanks Air Canada). They didn’t care much. We got the test results by email in about 2 hours.

We bought 3 T-Money cards from the Transportation Center for 4000 won each (later I learned that we overpaid - standard T-Money cards are 2.5k at convenience stores. However, we got Tourist Cards, which have special benefits, so it’s worth the extra cost if you go to some specific places. Do some research!), and got on the AREX train to get to Seoul. Transferred to the 1 train, they have cool conveyor belts to help you move heavy luggage if you’re tired after a long flight and the airport. Got to hotel, checked in, with negative entry tests, and felt great because we could finally be assured our vacation could start with no issues!

All in all, entry was super smooth. Just make sure to do your K-ETA and Q-Code and you should be fine.

Trip

Night: Cheonggyecheon Stream, and got dinner at Burger King because nothing else was open. This was a lesson - a lot of places in Korea don’t stay open late past like 20:00, and they don’t open until late too, until 8:00 to 10:00.

July 12: Morning: Hyundai Dept Store Seoul, I-Seoul-U sign. Afternoon: Myeongdong Shopping Street, DDP. Holy crap what a day, I was awfully tired. Noodles for dinner on tired feet was a good feeling. Not much to say, shopping and architecture, other than Myeongdong being destroyed by COVID. Oh, and we picked up our SIM card here. It’s very far from the airport, but the benefits are unlimited call/text/data for just like 29 USD. We got the Lite plan - 300mb of fast data, but the “slow” data is still 3mbps, good enough for most everything.

July 13: Morning/afternoon: Starfield COEX Mall/Library (pronounced Liberry by my sister). Night: Lotte Tower Rained on! Went back to the hotel and found a local museum to go to, walked around some shops. COEX was pretty fun! We ran around the place buying things, and although most of the food places were closed, was pretty fun. Dad bought a lot of kitchenware to take home.

July 14: Morning: Gyeongbukgung Palace. Afternoon: Hanok Village (construction! pretty hard to explore, gave up after not being able to find a good way in). Night: Naksan Park. Probably one of the prettiest views of the trip, except the mosquitoes were outrageous. Everyone was flailing around, it was pretty funny, but the view was worth it. I got bit 3 times, while wearing a jacket and jeans; my dad and sister fared worse. Another thing you’ll learn in Korea is there’s a million couples everywhere in Seoul. Everywhere is couples. My sister and I got mistaken for one on more than one occasion. Gyeongbukgung doesn’t actually begin selling tickets until 9, but going a bit early allows you to get super nice pictures of the first courtyard area.

July 15: Morning: War memorial, pretty solemn. Afternoon: Goto mall (giant thing), Gangnam. Night: Banpo Bridge, except we waited 3 hours and the show never happened. We were confused, everyone else there was confused, and after the last showing was meant to happen we just left. It was still fun though, because there were some nice people singing and having fun at the center circular stage.

July 16: Morning: Hybe Insight. Afternoon: National Museum of Korea. If you’re going to HYBE, everyone needs their own smartphone, even the children. But if you are missing one, you can borrow one of theirs.

July 17: Morning/Afternoon: Visited a friend down in Suwon. Night: N Seoul Tower. Booked tickets on Klook, they’re cheaper by like half! N Seoul Tower was pretty.

July 18: Morning: Flight to Jeju Island via Gimpo. This is where our planning starts to go weird. Afternoon/Night: Yongduam/Yongyeon Park/Rock. In Jeju, we didn’t have an international drivers license or anything, so taxis are key (get your own car if you’re planning on going to Jeju if you can!). If you can’t speak Korean like us, taxi drivers rarely understand english, so you will either have to pull up the place you want to go to in Korean or, to save both time and effort, use a booking app like Kakao T. It’s fast, you don’t have to flag a taxi and can just call it, and most importantly, you can book your destination on the app. This app and Naver carried our trip.

July 19: Morning: Arte Museum. Afternoon: Dodu-dong Rainbow Coastal Road + messing around on the rocks. This day was so relaxing. The Arte Museum is a bit of a pain to get to if you don’t have your own car. We ended up taking a taxi, as the bus would have taken 90 minutes. Also be aware of buses that have the same number but split later on - they announce the split in Korean, which we don’t understand, and this led to confusion and big time loss once.

July 20: Morning: Flight to Busan. Afternoon: Oryukdo Skywalk. The skywalk itself isn’t as special as I thought it would be, but the area is so pretty - there’s trails carved into the side of a small mountain (pretty view) and the cafe near the skywalk has a pretty view. The water misting cooling devices were also a highlight, funny enough. My sister loved them

July 21: Morning: Bujeon Market. Afternoon: Songdo Cable Cars, UN Cemetery, Magnate Cafe. Apparently the Magnate Cafe is owned by the father of some kpop artist. The more you learn i guess. Great smoothies though. Bujeon Market was epic, but any of the markets will give you a similar experience. It’s the feeling of being there and seeing an airport sized indoor farmers market of everything under the sun that gets you. And there’s not just one in the city, but quite a few! and Bujeon isn’t even the largest one!

July 22: Morning: Train to Ulsan. Afternoon: Daewangnam Park + Lighthouse, Ilsan Beach. We caught Ulsan just in time for its annual shipbuilding and sea festival (July 22-24). Ulsan is an industrial powerhouse, home to a lot of Hyundai and pretty views. Daewangnam contends for best view of the trip.

July 23: Morning: Amethyst Cavern Park + Temple + Zoo. Afternoon: Petroglyphs in Cheonjeon-ri. Probably one of my favorite days of the trip. Gosh, Amethyst Cavern was pretty good. Not much english signage but the mountains were pretty, the view was pretty, the caves were pretty, etc. The petroglyphs hike was also fun, and the temple was very interesting, although to be honest we were slightly intimidated by the scale and tone of it. We had KyoChon chicken for dinner. Much recommend. Lots of chains in Korea, get it sometime.

July 24: Morning: Train to Incheon (actually, Seoul and then train to Incheon) Afternoon: Songdo Central Park. Songdo Park at night is very beautiful. You can tell everyone living around loves it too

July 25: Morning: Shrimp Tower, Chinatown. Afternoon: Some kind of supercomplex Homeplus + E-Mart + Mall all in one. Yes, we went to go see a 2 story tower with absolutely nothing special with the exception of being shaped like a shrimp. In Korean it’s called 소래포구 새우 타워. Go see it. It’s funny.

July 26: Morning: Local walk, parks. Afternoon: FLIGHT!!! We brought luggage onto a bus obviously not meant for carrying luggage. This is when I was glad that we packed light, because our small carry ons slipped into the footwells and all was well

July 27: Home (and here I am)! I won’t write much here other than that Vancouver transfer security is horrid, and that leaving Incheon was very smooth. All the food in Incheon is in floor 1, B1 (mostly B1), or the Transport hub. The US has literally no entry requirements for vaccinated travelers, and I know you guys don’t want to hear about country specific exit requirements, because it won’t even apply to you. God help Canadian airport wait times. Please. Open another lane. I beg. Half the people in my line were late for connecting flights because of you :grr:

Final Remarks

The tourist places were pretty well populated, although not by tourists, even though I saw a few milling around the popular places in Seoul. Most places (and all in places like Ulsan) were mostly filled by domestic tourism, which seems to be booming. Almost everything is up and operating just fine, although you’ll occasionally find the shuttered place. Be sure to use Naver for directions. Google maps just doesn’t cut it for the reasons above, although it can be good for searching for tourist places in english (and fast food, which you can’t search for generally in Naver)

If any of you have questions about my entry experience or the rest of the trip, feel free to leave them below and I’ll try my best to respond to it! I heard that some entry requirements have changed in this short 2 week timespan, like now you have to upload your post arrival negative covid test to Q-Code.

A lot of u/Sybernova11’s post here explains public transportation way better than I did!

~this blog is WIP and so is this page~

Description

Initially made by Zerthox, Clearvision is a project created by myself and other members of the ClearVision organization, or the ClearVision Team. It is designed to customize discord, a chatting platform.

It’s a theme written in scss and hosted on github. You can install it via any discord injector that supports .theme.css files, such as BetterDiscord, Powercord, Vizality, Replugged, etc etc.

Support

You can find me and the rest of the team at our support discord, or just open an issue on our repo.