A mathematical space is, informally, a collection of mathematical objects under consideration. The universe of mathematical objects within a space are precisely defined entities whose rules of interaction come baked into the rules of the space. A space differs from a mathematical set in several important ways:
1) A mathematical set is also a collection of objects - but these objects are being pulled from a space (or universe) of objects where the rules and definitions have already been agreed upon. In other words, a space is required to think about sets.
2) A mathematical set has no internal structure, whereas a space usually has some internal structure. This could mean a variety of things, but typically it involves interactions and relationships between elements of the space, as well as rules on how to create and define new elements of the space.
In Bayesian Inference, the idea of a hypothesis space is very important, which is the set of hypotheses under consideration.
Examples of Mathematical Spaces
One motivating example is the idea of “space” itself, which is best modeled by 3 coordinates (each a real number) representing a location in space. This may not be a perfect representation of the space we live in, but it’s a mathematical model that has proven the most useful.
Simpler examples include the space of counting numbers, a boolean space (either true or false). You can also have a space of permutations of a rubik’s cube, or a space of continuous functions on [0, 1].
Constructibility
Typically, the rules of a space tell us how to construct values (or objects) in that space. For example, one can construct 2 very simple rules for defining the space of natural numbers:
1) 0 is a natural number
2) every natural number can be used to build another one by taking its successor.
By using the first rule and then the second rule successively, any natural number can be built.
For the Boolean or Binary space, it’s even simpler:
1) FALSE is a boolean value
2) TRUE is a boolean value
It seems like every space should have clear rules for construction. After all, if we’re going to talk about objects in these spaces, we’re going to need to precisely define what they are! It turns out that elements of one of the most common mathematical spaces, real numbers, are not in general constructible! In other words, there’s no finite set of rules that will allow us to define every number we might want, and there will always be some numbers that are forever undefinable.
One can get around this by considering a subset of real numbers, say rational numbers or algebraic numbers, but then when more values are needed there is a tendency to want to create more construction rules. Therefore, it’s often easier to just say “real numbers” and we know that means any creative rule that precisely defines the locations of a real number on the number line can be used.
Discrete vs Continuous Spaces
Intuitively a discrete space is made up of separate points where one needs to “hop” from one to the other without going through intermediate values. Examples are the natural numbers, finite spaces (like the binary space), or the permutations of a rubik’s cube.
A continuous space acts like real numbers where you can always find values in between any 2 existing values. The rational numbers are also continuous for the same reason.
Continuous spaces come equipped with a topology which defines how the points connect with and deform into each other. These are called topological spaces. A space must have a topology in order to be considered continuous or discrete - otherwise there’s no meaning to the term “in between”!
A discrete space can have a network topology which is a bit simpler. Every object in this space comes equipped with a set of “neighbors” as a directed graph. This is distinguished from the discrete topology which assumes that every point is an island and has no neighbors.
Some spaces can be a mixture between continuous and discrete.
Similar Concepts in Computer Science
The concept of a type in computer science, or a data type, is in many ways the practical implementation of a mathematical space.
In actual programming, all values are constructible. Any programming language that purports to have “real numbers” is using some constructible subset or approximation (like floating point numbers).
In object oriented programming, a class or struct is also a data type and represents a space.
Creating New Spaces From Existing Spaces
There are many ways to create new spaces from existing spaces. Here are a few examples:
A subspace is a space restricted to only a certain set of points. In some cases this can simplify problems, and in other cases the structure of the subspace with its gaps can actually be more complex and interesting than the original space.
One can take the sum space of two spaces A and B (A + B), with the following construction rules:
1) if x is a member of A, then A:x is an object in the space
2) if x is a member of B, then B:x is an object in the space
Note that each element is tagged with the space it came from. This may or may not be necessary to avoid naming collisions depending upon the situation.
One can take the product space of two spaces A and B (A * B), with the following construction rules:
1) if x is a member of A and y is a member of B, then (x, y) is a member of the product space
One can take the map space of two spaces A and B (A —> B, or B^A) with elements being a map of every element in A to an element in B.
Probability Distributions over Spaces
In Bayesian Inference, we usually consider a hypothesis space and want a probability distribution over that space. In discrete (and particularly finite) spaces, probability distributions are easy to define. In continuous spaces, one might need to construct a probability distribution function but the underlying rules on what is allowed come from measure theory (and are called probability measures)