AI feels distant until its wheels roll through our lives. Then the pain becomes personal.

That is my way of describing a feeling, not a verified quotation from a famous person. We used to discuss AI as a future event. Now it is changing how we work, what knowledge is worth, and how people demonstrate their abilities.

But another possibility concerns me more: what if AI brings not just efficiency, but treatments, materials and discoveries that did not exist before? How would we reassess its risks? Someone considering an abstract technological danger may answer very differently when facing only a few months of life.

1. When discoveries outpace understanding

Recent mathematical announcements make this question concrete. On September 8, 2026, OpenAI published a result on the Navier–Stokes Millennium Prize Problem, reporting a proof from an internal system and releasing a paper and Lean formalization.[1] On October 6, it released a further collection of mathematical results and formalizations, later announcing support for mathematicians to examine and develop them.[2]

An institutional announcement, formal proof checking, human understanding and the mathematical community's assessment of significance are separate things. A release does not establish that every question has been settled. Equally, incomplete human understanding does not automatically make a result false.

We may increasingly obtain useful results before developing a full understanding of them. Can verification keep pace with generation? If proposing an answer becomes much cheaper than checking it reliably, the bottleneck shifts toward deciding which answers deserve trust.

Mathematics at least offers relatively well-defined routes to formal verification. Drugs and materials must also survive contact with the physical world. A predicted structure is not necessarily manufacturable; a promising molecule is not yet a safe treatment. Discovery, validation and application remain different stages.

2. Where does the “would you take it?” example come from?

I remember an extreme hypothetical: you have a terminal illness, and AI offers a treatment humans do not fully understand. Trying it leaves some hope; refusing leaves none. Would you accept it?

The closest source I could trace is Geoffrey Hinton's February 20, 2020 thought experiment about a black-box surgeon. His hypothetical gives the AI a higher cure rate than a human surgeon, despite its inability to explain how it works, and asks whether that opacity justifies prohibition.[3] It concerns explainability and outcomes, not an existing cancer drug that nobody understands.

The terminal patient and unknown medicine in this essay are therefore my extension of the question, not Hinton's original wording or a clinical case. In particular, “no hope without it” belongs only to this deliberately extreme hypothetical. It does not describe advanced cancer in general.

Even within that hypothetical, I find it difficult to condemn someone who wants to try. The loss confronting them is already tangible; the possible benefit represents their remaining future. A healthy person asking for more time to validate a treatment and a dying person asking for earlier access occupy profoundly different positions.

Yet the story hides a premise: where does the claimed hope come from? Reliable evidence, or a model's confident assertion? Incomplete mechanistic understanding despite evidence of benefit is different from an absence of evidence that something works. The first cannot justify the second.

This also gives me pause: when AI becomes powerful enough to save all of humanity, it may also possess the power to destroy it.

3. Real AI drug research matters more than miracle stories

One verifiable example is rentosertib. A randomized, double-blind, placebo-controlled phase 2a study published in Nature Medicine in 2025 tested this candidate, discovered and designed with generative AI, for idiopathic pulmonary fibrosis. It enrolled 71 participants over a 12-week treatment period and reported findings warranting further study.[4]

This is not the hypothetical cancer drug. Nor does the study establish long-term efficacy or completion of all requirements for approval. Its significance is that an AI-assisted candidate reached clinical investigation, where patient data can test the promise.

Materials research has a similar boundary. In 2023, Google DeepMind reported roughly 2.2 million predicted crystal structures from GNoME, including around 380,000 more stable candidates for experimental exploration.[5] Those numbers represent expanded predictive search, not an equal number of mature industrial materials.

I want to see this work deliver, rather than count candidates as benefits humanity already possesses. Reproducible experiments, reliable treatments and affordable products will ultimately matter more to public trust than impressive predictions.

4. Do people ignore risk, or reorder it?

My first instinct was that human nature ignores safety when the reward is large enough. That is too absolute. A more defensible description is that personal benefits and the cost of waiting change how much risk people are willing to bear.

Delaying a technology also has costs. If a treatment eventually proves effective, patients who missed its benefits while waiting cannot disappear from the discussion. But a promise to save lives cannot exempt a treatment from validation. Otherwise, those with the strongest hopes become the most exposed to false promises.

I worry that concrete benefits and dispersed risks receive unequal attention. A patient returning to ordinary life is visible. An increase in the probability of accidents or misuse is harder to give a face. Decision-makers may underestimate that risk, or invoke distant dangers while neglecting immediate suffering. Both distortions matter.

The debate must therefore ask whose benefits and whose risks are being counted. Who supplies the evidence? Who explains failure? If the beneficiaries and those bearing the costs are different groups, a simple total of expected benefits misses a central issue.

5. Could tools for saving lives also help cause harm?

There is evidence behind this concern. The 2022 Nature Machine Intelligence paper “Dual use of artificial-intelligence-powered drug discovery” examined misuse of drug-discovery AI and demonstrated a computational proof of concept involving potentially toxic molecular candidates.[6]

It supports a limited but important conclusion: computational capabilities developed for therapy can potentially be redirected toward harm. It was not a demonstration of actual weapon production. Chemical risks at the molecular level also differ from biological weapons capable of spreading. Collapsing those distinctions into “AI can already manufacture bioweapons” would weaken the argument.

Real-world harm still requires resources, facilities and expertise. That does not make computational advances irrelevant. My judgment is that reducing a difficult research bottleneck should prompt a fresh assessment of the safeguards that remain. We need to examine how capabilities reach the physical world, beyond whether a chatbot refuses one dangerous question.

From a security perspective, boundaries must cover the whole chain: what information a model accesses, which tools it invokes, what experimental work it can request, who approves actions, and whether incidents can be traced and stopped. One refusal at the conversational interface cannot carry that entire burden.

6. One person's consent cannot speak for everyone

This is the distinction I consider most important. A patient making an informed choice about risks attached to their treatment is one question. Developing or deploying a capability that exposes uninvolved people to risk is another.

Consent to the first does not establish consent to the second. Choosing an experimental treatment does not mean accepting unrestricted use of the underlying technology in every setting. It certainly does not mean everyone else has accepted the consequences.

Likewise, mathematical success cannot certify a system's reliability in medicine, security or physical operations. Trust must attach to a particular task, body of evidence and boundary. It cannot expand indefinitely from one spectacular achievement.

I favor bounded application supported by evidence, independent validation, clear responsibility, continuing monitoring and permissions that can actually be withdrawn. Incomplete explainability need not prohibit every use. It also cannot excuse abandoning scrutiny.

7. When we depend on it, who can still ask us to wait?

Another risk may emerge after success. If AI makes us healthier and more productive while taking on essential work, the cost of suspending it rises. Even when new risks are discovered, people may continue using it because withdrawal has become too costly.

That is a concern, not a predetermined outcome. Preserving choice means acting before convenience becomes dependence: allowing external scrutiny, keeping systems replaceable, and retaining necessary human knowledge and practical skills. Otherwise, a series of locally reasonable decisions may leave us with very little freedom to decide later.

I cannot conclude that humanity will reject these risks, or that everyone will ignore them. People may accept more risk without every risk becoming acceptable. The answer depends on evidence, consequences, alternatives and who has the right to decide.

When AI offers a genuinely lifesaving option, the hardest question may only be beginning: can we sincerely want further progress while retaining the ability to inspect, constrain and, when necessary, pause its use?

I hope we will be able to say that we accepted a risk because we understood what we were accepting, rather than because we had run out of choices.

Sources and verification notes

This is an opinion essay with factual sources checked as of November 10, 2026. The opening wheel metaphor has no verified original attribution and is presented as a personal reflection. The terminal patient and unknown medicine are a thought experiment, not medical advice or a reported treatment case.

  • [1] [OpenAI: On the Navier–Stokes Millennium Prize Problem](https://openai.com/index/navier-stokes-solution/), September 8, 2026. This essay describes the institutional announcement without asserting prize recognition or completion of all independent review.
  • [2] [OpenAI: Sharing AI progress in mathematics](https://openai.com/index/sharing-ai-progress-in-mathematics/), October 6, 2026, including the October 15 update.
  • [3] [Geoffrey Hinton's original post](https://twitter.com/geoffreyhinton/status/1230592238490615816/), February 20, 2020. Direct retrieval was unavailable; the attribution was cross-checked through [InfoQ's contemporary report and original-post link](https://www.infoq.com/news/2020/03/explainable-ai-survey/) and [teaching material preserving a screenshot](https://ai.ethicsworkshop.org/Library/AI-Case-Disease.pdf). This essay paraphrases the thought experiment and does not attribute the unknown-drug scenario to Hinton.
  • [4] [Xu et al.: A generative AI-discovered TNIK inhibitor for idiopathic pulmonary fibrosis: a randomized phase 2a trial](https://www.nature.com/articles/s41591-025-03743-2), Nature Medicine, June 3, 2025.
  • [5] [Google DeepMind: Millions of new materials discovered with deep learning](https://deepmind.google/blog/millions-of-new-materials-discovered-with-deep-learning/), November 29, 2023.
  • [6] [Urbina et al.: Dual Use of Artificial Intelligence-powered Drug Discovery](https://pmc.ncbi.nlm.nih.gov/articles/PMC9544280/), Nature Machine Intelligence, March 7, 2022. The citation concerns dual-use potential, not experimentally validated weapons.