Sandesh's Personal Website
Writings about science and my life

Notes on Tao's Mathematics in the Age of AI

The following post consists of my notes from an essay by Terence Tao, Mathematics in the Age of AI.

While his essay focuses on mathematics, I feel some of the themes I describe below apply broadly to all scientific fields including physics.

The framing of this essay is phenomenal and I have to admit, only something a clear thinking mathematician could have accomplished. Rather than debating whether AI is great at mathematical reasoning or not today, let’s assume that AI is quite good one day. Then what are the goals and values of mathematical research? I think this is the first time I have seen values being tied to the idea of doing research.

Foundations of mathematics had to be built from primitive concepts such as sets in the first half of the 20th century. A consequence of this crisis in foundations and its resolution gave rise to mathematics which could be critically inspected and taught. A happy accident was the mechanization of the mathematical proof. A clear mechanical procedure emerged to examine if each statement followed from a previous one and was consistent with previous results and foundational axioms.

Tao believes that we are in a period similar to the early 1900s, a period of turbulence in what doing mathematics means. He defines the problem as follows:

This time, though, what is being stress-tested is not our foundational framework for mathematical truth, but rather the largely implicit framework of mathematical values and practices: what we consider a contribution to be, what we reward, what we regard as understood, and who — or what — we regard as having done the work.

Much of the tests on evaluating the strength of AI models is not carried out under controlled scientific conditions. Reporting bias leads to successes being reported whereas failures are not. Parameters such as human scaffolding, compute size and degree of contamination with prior literature are not disclosed.

What are the goals of mathematics? Tao lists the following incomplete list:

  1. to solve unsolved problems, both pure and applied
  2. to develop new theories, structures and techniques
  3. to understand the world around us
  4. to build and sustain a community of mathematicians
  5. to train the next generation of mathematicians, and to let them guide the future directions of the subject
  6. to contribute to the shared and cumulative network of mathematical knowledge
  7. to create enduring works of aesthetic value

Goodhart’s law as popularized by Strathern in relation to monetary policy: When a measure becomes a target, it ceases to be a good measure. Goodhart’s law might limit us on quantifying the skills of AI models.

Case study: Problem solving

Let’s say our goal is to correctly solve and verify as many mathematical problems as possible. If we let AI optimize this goal, a new failure mode can appear. What if the AI generates a long proof that nobody understands? Even if we ask AI to explain it to us, the AI generated explanation can fail to adequately place the result in prior literature or offer a high-level overview that a mathematician can grasp without a significant investment of time.

There is also the problem that even if the exposition can be optimized to be well-written, there is a tendency for AI to make it too well-written. A proof written by a human has natural friction: some steps are harder to digest while others are easy to follow. When a reader slows down at points of natural friction, their understanding grows as they grapple with what they don’t understand.

Proofs also need to be accepted by the community. The existence of a proof might help a mathematician in making progress towards an orthogonal direction, but only understanding, digesting and accepting a proof allows us to generate and tackle harder problems in the same direction.

Tao describes a beautiful problem-solving pipeline in Figure 4. This pipeline highlights that “impedance mismatches” can occur between various steps of the process. We will transition from an era of proof scarcity to proof abundance. Most of scientific institutions and their idea of a research program rely on this scarcity.

What can we do

Leiden Declaration on Artificial Intelligence and Mathematics lists 23 recommendations. Tao argues a very interesting point on the culture of proof generation:

I argue that we need to decrease the emphasis that our culture places on proof generation, and in particular on being the “first” to solve a problem, and correspondingly increase the emphasis we place on proof digestion: exposition, refereeing, publication, and canonicalization.

He says that even if a proof has been verified to be correct, until it is understood clearly by the author i.e. they can give a clear, expert-level seminar on the results, it should not be published.

Particularly in education, it might be appropriate at certain times to restrict AI use: as the goal of training a mathematician is not achieved by producing correct homework. In other areas, we must encourage and share innovative practices on AI usage. We should share in public discourse how AI is being used in research instead of holding it back to avoid judgement from our peers.