From DeepMind to OpenAI, from the U.S. to China, from autonomous driving to large language models, the past decade has been a nonstop parade of AI breakthroughs and billion-dollar funding rounds. Yet one uncomfortable theme persists: AI companies can’t seem to make a profit.

Notably, this problem is specific to standalone AI companies. Google, Meta, and ByteDance are swimming in cash, driven by AI-powered advertising engines and platforms.

Why the discrepancy?

Perhaps AI, as it stands, is less of a ‘money printing’ business and more of a lubricant for ‘existing money printers’. If you don’t own the money printer, you’re stuck in the lubricant business.

Efficiency Isn’t Enough; Go Direct

The argument often made by AI companies is that they “raise efficiency” in industries ranging from customer support to drug discovery. That may be true, but efficiency gains rarely translate into big profits for the software provider.

Take drug discovery: AI can speed up processes, help identify promising candidates, and optimize workflows. But many AI companies stop there, collecting service fees while the real money—the billions of dollars—comes from the drug itself. It’s a long, risky, and expensive journey, but only the final product captures the lion’s share of value.

The same is true in creative industries. Building AI as an add-on to existing platforms—like adding features to Adobe Premiere or JetBrains IDEs—limits your upside. These platforms own the customer relationships and dictate the terms, relegating AI startups to secondary roles.

Contrast this with AI-native solutions like Cursor, which integrates AI deeply into the coding process, or OpusClip, which reimagines video editing for social media creators. These tools don’t merely assist—they replace traditional workflows entirely, enabling direct customer relationships and scalable revenue models. The lesson is clear: being an add-on limits your upside. To capture real value, you need to own the entire process.

The Paradox of Disruption

AI startups love to claim they’re “paradigm-shifting.” But how can you disrupt legacy industries while relying on them for revenue?

When AI companies sell APIs or tools to legacy enterprises, the value of their technology is largely determined by how the customer deploys it. This creates two major problems. First, startups must invest heavily in customer enablement—helping clients integrate APIs, redesign workflows, and scale usage—just to ensure their tools are used meaningfully. Second, it exposes a deeper truth: if these legacy companies were capable of transformative efficiency, they wouldn’t need an AI startup to begin with.

The inefficiencies AI companies aim to solve—bureaucracy, slow decision-making, misaligned incentives—are the very reasons legacy companies struggle to adapt. Changing these organizations from the outside is a Sisyphean task. It’s not just a matter of technology but of overcoming entrenched processes and culture.

If AI is truly paradigm-shifting, then the only way to capture its full value is to remake industries from the ground up. Instead of being a lubricant for someone else’s money printer, AI companies need to own the printer. That means going direct to the final customer and delivering complete products, not just tools.

OpenAI’s move into consumer-facing products like ChatGPT+ hints at this shift. But even here, the monetization challenges remain steep. Subscription fees are a start, but the scale of revenue doesn’t compare to advertising-driven giants or companies selling physical goods.

AI, in its current form, is undeniably powerful, but power alone doesn’t pay the bills. To go from lubricant to printer, AI companies need to rethink their business models and aim higher. Otherwise, they risk becoming the oil in someone else’s machine—useful, but ultimately commoditized.

Challenges of Owning the “Printer”

Building an AI-first business is no small feat. It requires deep expertise, substantial capital, and the patience to navigate long, costly development cycles. But this effort is essential—not just for profitability but for uncovering AI’s true potential.

Consider drug discovery: while AI has excelled in preclinical drug discovery, the clinical stage—where costs rise by orders of magnitude—remains largely untouched. This disparity isn’t just a failure of ambition but a reflection of structural challenges. AI startups gravitate to preclinical work because it aligns with their strengths in data-driven modeling, while clinical trials require navigating patient variability, logistical complexity, and regulatory oversight—domains where AI’s impact is nascent.

However, efforts are emerging. Companies are starting to optimize trial design and patient recruitment. Still, these represent early steps in what must eventually be a larger movement: AI companies owning more of the pipeline to capture the true value of their innovations.”

Takeaway: Rethink the Premise of SaaS

The success of SaaS lies in its simplicity: lower costs for delivery and maintenance compared to traditional on-premise software. This model thrived during the high wave of SaaS adoption, where companies replaced existing software with more agile, scalable solutions. But the keyword here is existing—SaaS succeeded by slotting neatly into pre-defined workflows, offering immediate efficiency gains without disrupting the overall structure.

AI, however, doesn’t follow this playbook. Injecting AI into existing workflows often creates friction instead of seamless integration. AI’s transformative potential clashes with legacy processes that were never designed for it, making it difficult to find a “neat slice” where AI can plug in without complications.

If one truly believes this wave of large pretrained models—or even AGI—is as transformative as claimed, then the logical conclusion isn’t just to enhance existing workflows but to remake them entirely. This means designing processes from the ground up with AI as the central driver, not an auxiliary add-on. It’s a shift from retrofitting AI into old paradigms to building AI-native workflows that maximize its potential.

In short, the AI revolution isn’t just about technology—it’s about reimagining the very fabric of how we work and live. For startups, the challenge isn’t to fit into the SaaS mold but to break it.

In 2023, the world of artificial intelligence saw a major leap with the rise of large language models (LLMs) like GPT, sparking excitement among scientists keen to tap into their potential. Yet, as discussions unfold, there's a realization that equating large models solely with LLMs is a bit limiting.

Language, being textual, is one-dimensional, but the realm of science is awash with data that defy such simplicity. Take chemical bonds, often depicted in 2D graphs to represent their structure, while molecules themselves exist in the 3D realm. Then there's the even more complex n-dimensional wavefunction, critical in determining material properties like bandgap or thermodynamics.

After all, the essence of a 'large' model lies in its ‘large’ number of parameters and the ‘large’ amount of unlabeled data used in pre-training, principles that are by no means exclusive to language. So what are the use-cases of large models in science that might bring about real value? Here is a subjective list of what I believe could emerge in 2024.

I. Expert LLM for Scientific Documents

Current Status

While tools like ChatPDF defined the user experience of interacting with scientific literature using chat, there's a world of difference between a tool that's "barely functional" and one that's "actually useful." The current crop of LLM PDF readers falls short :

• either ignore all images in a document

• or employ rudimentary image-to-text preprocessing techniques (which lost lots of crucial information).

However, for an LLM to truly serve the scientific community, it needs to go beyond this simplistic approach.

Challenge

The images in scientific papers and patent files are not just filler; they often contain essential information that complements the text. This is especially true for fields where visual data, such as Markush molecule graphs in chemistry, play a central role in conveying complex ideas. The challenge lies not just in recognizing these images but in aligning the information they contain with the textual content of the documents. Representing scientific images in a way that an LLM can understand or retrieve and integrate with textual data is a puzzle that researchers are just beginning to piece together.

Moreover, the urgency to enhance how LLMs handle scientific images stems from an ambitious vision: we aim to evolve LLMs into advanced search engines. Such engines would enable us to search for specific entities like a Markush structure and retrieve all mentions of it across scientific papers and patents. This capability would drastically improve the efficiency and precision of scientific research and intellectual property management.

2024 Outlook

1. Expert LLM that can process, preserve scientific image and align image info with text

2. LLM system that can aggregate vast scientific literature and ‘search’ precise information

II. Large Models with Lots of Labeled data

Current Status

The development of Large Models like AlphaFold2 has been a huge milestone for science. However, despite its impressive capabilities, AlphaFold2 still cannot replace cryo-EM. For example, its utility in directly facilitating processes like molecular docking—a method used to predict the preferred orientation of one molecule to a second when bound together to form a stable complex—remains limited. Various review studies have highlighted these limitations, and translating these predictions into actionable insights for drug discovery and other applications still faces hurdles.

Challenge

Despite the leaps in computational methods, the nuanced complexities of biological systems often require experimental confirmation. This reality underscores a broader challenge in the field: high-quality, experimentally derived data is not only expensive to obtain but also difficult to scale. While computational models like AlphaFold2 provide invaluable tools for hypothesis generation and preliminary analysis, the lab bench remains irreplaceable for the foreseeable future. The future, however, holds promise for a more integrated approach, where computational predictions and laboratory experiments converge more seamlessly. Robotics labs, equipped with automated and autonomous systems capable of designing and conducting experiments, validate computational predictions at a scale and speed previously unimaginable. This integration could extend to employing Bayesian optimization and other advanced statistical methods to refine computational models based on experimental outcomes, creating a dynamic feedback loop that continually enhances the accuracy and applicability of predictions.

2024 Outlook

1. More research institute and companies looking at the DeepMind / Berkeley Lab approach of ‘computation guided experiment design’ + ‘autonomous experiment design, execution and evaluation’

2. More open-sourced scientific database become available beyond PDB, OC20 etc.

3. Based on #2, the team that can ‘clean’ the most amount of open database will gain visible advantage. This is dirty work but highly necessary.

III. Large Models with Few Labeled data

Current Status

In the vast majority of scientific fields, such as drug discovery and materials science, the reality is starkly different from the ideal scenario of abundant, labeled data. For instance, in the realm of ADMET (Absorption, Distribution, Metabolism, Excretion, and Toxicity) in drug discovery, obtaining even a few hundred data points can be a rarity. Similarly, in materials science, exploring a new formula necessitates the actual creation and testing of a material in the lab, a process that is not only costly but often impractical. Given this scarcity of data, traditional direct learning models generally underperform due to their reliance on large amounts of high-quality labeled data.

This is where pre-training would shine —— first pre-training on unlabeled molecular structure data, followed by fine-tuning on sparse labeled data for specific downstream tasks. This approach has begun to show promising results in areas like QSAR, indicating a potential pathway to overcome the data scarcity challenge.

Challenge

The challenge of pre-training in the virtually unlimited chemical space is daunting. Identifying sources of high-quality data, deciding on the most effective way to represent molecules (be it as graphs via Graph Neural Networks (GNN) or as 3D conformers), and figuring out how to efficiently conduct multi-task training are all significant hurdles.

Multi-task training, in particular, is complex in this context due to the diversity of data sources and formats. For example, using Density Functional Theory (DFT) as training data introduces variability because different DFT methods can yield different data formats depending on the choice of base functions. This variability complicates the model's ability to generalize across tasks and datasets, making the development of versatile and effective models a challenging endeavor. Despite these challenges, early work in multi-task learning is paving the way for exciting developments.

As we look toward 2024, the potential emergence of large pre-trained science models, particularly in materials science, holds great promise. Such models could revolutionize fields like catalysis, battery development, and polymer science by enabling more efficient exploration of new materials and processes.

2024 Outlook

1. In material science, people will figure out how to run multi-task learning so that a unified large model can process data from various experimental tools (x-ray, stem, afm etc.) and different kinds of computation.

2. drug design and batteries are likely to benefit first from this new technology, given their fierece competition and thirst for innovation

Summary

In short, the exploration of large models in science reveals a future far beyond their initial application in language processing, promising a transformative impact across numerous fields.

Looking ahead to 2024, we might witness advancements that will showcase the full potential of large models in scientific discovery. The narrative of large models is expanding, and the coming year promises to deliver compelling evidence of their broader significance in the scientific community.

#LargeScienceModel #AI4Science

❌ Because Bigtech are benevolent overloads? Nope. Bigtech are for-profit and are 'legally' required to not waste shareholder's money. Also, Bigtech is an institution with institutional goals. Even if a CEO is 'benevolent', under a market society, an institution HAS TO profit and grow in order to attract talent. Otherwise, people leave and the company falls apart.

❓ Because Bigtech need these 'vanity projects' to attract talent. True but trivial. The ground truth is still that "Bigtech are for-profit institution". If open-source is just for appearance, it would be impossible to justify the billion dollar price tag. No CEO would put his neck on the line for this since the ROI cannot be reasonably measured.

✅ Because Bigtech need these to SURVIVE competition. If you are Google, would you build your future project map on top of OpenAI, which Microsoft 'controls'? Unlikely. So now you have two choices: to build another close-sourced alternative; or to build an opensourced one. The beauty of open-source is that, if run well, it can enlist the contribution of a much broader community so that you don't have to fight Microsoft/OpenAI alone.

✅ Because Bigtech spending billions helps justify their trillion dollar valuation. The billions Meta invested in MetaAI (Pytorch, Llama etc.) is not only justified but a matter of life-or-death, because Meta is a public company and its valuation premium (p/e ratio) depends on investor's believe that it will continue to be a supreme tech leader. If Meta is just a social network company selling ads, it would be worth only 1/3 of what it is today.

There are other considerations out there. For example, a company developing powerful open-sourced software can sell enterprirse version and provide services deploying or maintaining it. This is actually how Linux/IBM makes money.

Open-source is not some utopia or magic pills. It is a well-thoughtout strategy on the forefront of tech, business and finance. Companies like Databricks, Hugging Face etc. are very interesting cases to study.

You never know when or where AI and Science would converge

1/ What’s new with AlphaFold3

AlphaFold was known for cracking the ‘holy-grail’ challenge of protein folding — aka, given a protein’s sequence (1d), can you predict its 3d structure. Now, its latest update claim to go beyond protein and can now also predict ligand-protein interaction, which is a critical step in drug discovery. it also claim to be able to predict nucleic acids, which would be quite relevant for cutting edge healthcare innvoations such as mRNA vaccines.

Context: as a computational method, alphafold2 was considered a Nobel Prize worthy achievement, because it matches, for the first time, the accuracy of experiment. Which means the cost of biology research just dropped by orders of magnitudes. However, for drug discovery, understanding the structure of protein is just the first step. The next step is understand “if i put a drug on the protein, how would the drug work”. The update of AF3 is to address this.

2/ Eric Schmidt Launches new AI for Science initiative

This new research institute is headed by Andrew White of University of Rochester, and will be focusing on AI’s application in biology as a start.

Context: Eric Schmidt has been a strong advocate for AI for Science, having previously pledged 140 million USD for AI for Science scholarship. He also gave a great speech at MIT on why AI is an inevitable path for future science research. https://www.technologyreview.com/2023/07/05/1075865/eric-schmidt-ai-will-transform-science/

3/ BioMap secures contract with Sanofi worth up to $1bn

BioMap, co-founded by Robin Li (founder of Baidu) in 2020, secured a contract from Sanofi to develop advanced AI models and protein Large Language Models that will enable biologics design and multiparametric optimization.

Under the terms of the agreement, BioMap will receive an upfront cash payment and near-term payments for reaching module development milestones from Sanofi. BioMap will be eligible to receive payments of over $1 billion based on achievement of pre-clinical development, clinical development, regulatory, and commercial milestones.

Context: Sanofi is arguably the most active proponent of AI in drug discovery, having previous announced that it is ‘all-in’ on AI. Baidu and Robin Li is very active in the crowded LLM space. Using LLM to do biology research has been a heated debate, as whether language is the ultimate representation of knowledge is still unclear. Baidu’s LLM however, is met with mixed review (at best) when launched.

ChinaTalk

How ERNIE, China's ChatGPT, Cracks Under Pressure

ERNIE Bot, Baidu’s widely anticipated LLM chatbot, became publicly accessible on August 30, after five months of invite-only mode. Now, anyone in the world can play with a Chinese LLM. The move, as MIT Technology Review reports, is likely the result of Baidu gaining regulatory approval — meaning that Chinese regulators are reasonably confident about ERNIE’s performance, safety parameters, and political allegiance…

Read more

3 years ago · 31 likes · Irene Zhang

4/ Insilico Gets $80m upfront for AI-designed drug

Under the terms of the agreement, Insilico granted Exelixis an exclusive, worldwide license to develop and commercialize ISM3091, and other USP1-targeting compounds, in exchange for an upfront payment to Insilico of $80 million anticipated in the third quarter 2023. Insilico is also eligible to receive future development, commercial, and sales-based milestone payments, as well as tiered royalties on net sales. “ISM3091 is the third clinical-stage program made possible by Chemistry42, Insilico Medicine’s generative AI platform for small molecule drug discovery,” said Alex Zhavoronkov, Ph.D., founder and CEO of Insilico Medicine.

Context: Insilico is one of the most active AI Drug Discovery startup globally, who also recently filed for IPO. Its 2022 revenue is $30m, which mean its 2023 growth would look quite well with this new deal.

5/ Meta demoed real-time decoding of images from brain activity

Using magnetoencephalography (MEG), a non-invasive neuroimaging technique in which thousands of brain activity measurements are taken per second, Meta showcase an AI system capable of decoding the unfolding of visual representations in the brain with an unprecedented temporal resolution.

Context: despite the struggles last year (VR failure, layoffs, market cap crash, executive leaves etc.), Meta is actually doing suprisingly well in the AI war, with Llama, Segament Anything, etc. Major respect to LeCun.

6/ ExxonMobil and Gerogia Tech: AI Predictions Could Dramatically Reduce Energy, Water Use in Oil Refining

A new kind of polymer membrane created by researchers at Georgia Tech could reshape how refineries process crude oil, dramatically reducing the energy and water required while extracting even more useful materials. They also have created artificial intelligence tools to predict the performance of these kinds of polymer membranes, which could accelerate development of new ones.

Context: the initial separation of crude oil components is responsible for roughly 1% of energy used across the globe, so to save of energy here will have a big impact. Beyond this usecase, material science actually plays a fundemental role is most aspects of our life, such as turning sea into fresh water. But developing a commercial viable new material is extremely challenging, more so than developing a new drug. As an example, despite the trillions dollar worth of EVs, and the urgent need for innovation, the battery material system we use is still not very different from the 1990s. Many is turning to AI to expedite new material discovery. CATL for example, has been quite active in adopting AI for new anode materials research https://onlinelibrary.wiley.com/doi/abs/10.1002/adfm.202303936 .

7/ CEDER Groups new research unveils a key feature of Solid-state battery using AI

Soft clay-like Li-superionic conductors, integral to realizing all-solid-state batteries, have been recently synthesized by mixing rigid-salts. Here, through computational and experimental analysis, researchers clarify how a soft clay-like material can be created from a mixture of rigid-salts.

Battery functions on atomic level. Usually simulating it accurately requires quantum mechanics compuation (such as DFT-MD), which are expensive and slow. DFT-MD without a HUGE supercomputer, will struggle to simulate even 1000 atom for 0.1ns. In this work, a deep learning neural network is trained to subsitute the DFT part and the DL-MD simulated 10,000 atoms for 2ns, at DFT accuracy.

Context: Gerbrand Ceder is one of the most known name in the battery world. He is also the Samsung Distinguished Chair in Nanoscience and Nanotechnology Research at UC Berkeley. He started the Material Project, analogous to the human genome project, which among other things, created a huge database of material structure and properties (think of this as the ImageNet of material science). Almost all AI companies in the material science space uses this work one way or another.

Disclaimer: This page contains links and summaries of articles that have been curated from various sources. The curator makes no claims as to the accuracy, completeness, or reliability of the information provided in the original articles. Links are provided for reference and convenience only. Readers should evaluate the source and credibility of any content referenced in this newsletter and proceed at their own discretion.

Probably not.

Because it's not just about the individual blocks; it's about how you put them together. The magic happens in the connections, in the way one block sits atop another, creating bridges, towers, or even castles.

That's a lot like bee and its dance. We might know everything about a single bee, but when lots of bees come together, something special happens. They begin to dance in ways we couldn't have guessed by looking at just one bee. It's not about what's going on inside each bee; it's about how they work together. Their dance, like your amazing block structure, comes from the interactions of many, many bees.

This is what scientists call "emergence." It's like a surprise party: you might know all the guests, but you can't predict the fun and memories that'll emerge when everyone's together. In the world around us, whether it's bees dancing, blocks connecting, or people laughing together, the true magic often lies not in the pieces themselves but in how they come together.

Simply put — it's the inter-action, not the inner-action, that makes the magic happen.

Emergence in Technology

The concept of emergence isn’t limited to nature. Today’s most groundbreaking innovations often spring from the interactions of simple elements. Let’s dive into some modern examples:

1. Silicon to Supercomputers: At the heart of every computer, smartphone, or tablet is a microchip. This chip is made of billions of tiny electronic switches called "transistors" that can either be on or off, much like a light switch. Individually, these transistors can't do much. But when billions of them start "talking" to each other, sharing information and making lightning-fast decisions, they give rise to the amazing computing power we see today. From playing video games to exploring virtual worlds, all emerge from these billions of tiny on-off switches working in harmony.

2. Elementary Math to AI Masterpieces: Think of artificial neurons in neural networks as students in a classroom, each doing simple math problems. Alone, a student solving a math problem is straightforward. But imagine a billion students working together, sharing their answers, and building upon each other's work. Suddenly, we've moved from basic arithmetic to advanced calculus. This is how systems like GPT function. Billions of artificial neurons, each doing simple math, come together to understand and generate human-like text. It’s emergence in action, turning simple math into thought-like processes.

3. Buyers and Sellers to Global Economies: Every time someone buys a toy, a book, or even a cup of coffee, they're participating in the economy. On their own, these transactions are simple and direct. But when you consider billions of these transactions happening worldwide, they give rise to intricate economic systems, stock markets, and global trade networks. These systems are so complex that even the brighest minds in the history could not predict (even Isaac Netwon lost money in the stock market). What might seem like a simple purchase can influence economic trends, stock prices, and even international relations.

4. Individual Posts to Social Movements: Think about social media. A single tweet or post may seem inconsequential. But when millions resonate with that sentiment and start sharing and posting about it, massive social movements can emerge. Movements started with individual stories and feelings but rapidly grew into global calls for change, reshaping societal discussions and policies.

5. Simple Rules to Complex Games: Consider the video games children and adults enjoy. The graphics, stories, and experiences in these games emerge from lines of code, each one simple and directive. But when millions of these lines of code interact, they create expansive virtual worlds, intricate storylines, and engaging gameplay. It’s not just the code, but how they relate and work together that brings the game to life.

The magic of emergence lies everywhere. Whether it’s the natural dance of bees, the intelligence of AI, or the pulse of global economies, it all boils down to simple components interacting in complex ways. When individual pieces come together, they create patterns, systems, and innovations far greater than the sum of their parts.

Emergence in Science

Emergence is more than just a fascinating concept; it's a pivotal shift in how we approach understanding the universe.

For millennia, scientists have meticulously dissected the inner-workings of entities, from the atomic structures of materials to the biological mechanisms of organisms. This pursuit of reductionism—breaking things down to their most fundamental parts to understand them—has borne fruit in countless ways. It's the reason we understand the periodic table, the human genome, and the principles of classical physics. The Large Hadron Collider (LHC), one of the most ambitious scientific endeavors, embodies this approach, diving deeper into the subatomic realm than ever before.

However, there's a growing sentiment that we may have hit a kind of intellectual bedrock. Despite the colossal investment and expertise poured into projects like the LHC, there are diminishing returns in our discoveries. While understanding the inner-actions remains crucial, the tide is beginning to turn, emphasizing the importance of interactions and emergence.

Why? Because emergence encapsulates phenomena that can't be understood merely by examining individual components. It pushes us to think about systems, relationships, and networks, rather than isolated entities. As we stand at the frontier of scientific discovery, the focus is shifting from the "notes" to the "symphony," from isolated particles to intricate patterns.

One of the most captivating illustrations of emergence in modern physics is the phenomenon of superconductivity. At certain temperatures, specific materials transition into a superconducting state, conducting electricity without any resistance. This isn't just about individual atoms or electrons behaving differently, but a collective, emergent behavior arising from their interactions. The phase change into superconductivity remains one of the grand puzzles of condensed matter physics. Despite our deep understanding of atomic and subatomic particles, the holistic nature of this transition defies a purely reductionist explanation, further underscoring the importance of studying emergent behaviors in science.

If the previous 3000 years was marked by a quest to understand the smallest building blocks of matter, the 21st century might very well be defined by our journey to understand the vast, interconnected dance of emergence. It reminds us that sometimes, to truly understand the universe, we need to step back and appreciate the grand choreography of elements coming together, creating a spectacle greater than the sum of its parts.

As we forge ahead, it's this balance between the microscopic and the macroscopic, the inner-action and the interaction, that will shape the next era of discovery. Embracing emergence doesn't negate the achievements of reductionism; instead, it adds another layer of complexity and beauty to our quest for knowledge. In this dance of understanding, both the solo and the ensemble play indispensable roles.

You never know when or where AI and Science would converge

§1. Machine Learning and Intelligent Systems

1/ OpenAI Just Bought a Game Studio

OpenAI announced the acquisition of video game startup Global Illumination, creator of the open-source "Minecraft" clone "Biomes." While OpenAI's motivations are unclear, the move allows them to bring aboard Global Illumination's talented co-founder Thomas Dimson, known for developing Instagram's content ranking algorithm. Speculation is that OpenAI could be looking to train AI models like GPT-4 in 3D virtual worlds resembling real life, release an AI-powered game as a new revenue stream, or use the technology to generate dialog for video game NPCs. Overall, the acquisition suggests OpenAI sees potential in applying its AI capabilities to the gaming industry, but the company's specific plans and expectations remain unknown.

2/ Dialpad Released The State of AI at Work Report

Dialpad — a industry-leading Ai-Powered Customer Intelligence Platform — today announced findings from its 2023 Dialpad The State of AI at Work Report. Here is a summary of the key findings from the Dialpad report on the state of AI adoption in customer service and sales:

• ChatGPT is driving new interest in AI, with 76% of respondents saying they will now consider AI tools after using ChatGPT when they previously did not use AI.

• 79% of sales and customer service professionals using AI have seen a positive impact on their job performance. Media, entertainment, defense, software, and pharmaceutical industries have been early AI adopters.

• 70% are not concerned AI will take their jobs. 84% of sales managers believe AI tools are crucial for growth. But 37% cite lack of budget as a barrier to adoption.

• 84% say their company lacks an AI policy. Over half believe AI is not yet accessible. This highlights the need for ethical guidelines as adoption increases.

• 30% of small businesses with under 50 employees are most skeptical about using AI tools.

• Overall, the report shows growing positivity towards AI among customer service and sales workers, but barriers around budget, ethics, and accessibility remain. As adoption spreads, developing responsible and inclusive AI policies will be important.

3/ US Court Rules GenAI cannot receive copyrights

A U.S. federal court has ruled that works of art created solely by artificial intelligence without any human input cannot be copyrighted under current law. The decision came in a case where computer scientist Stephen Thaler applied for a copyright on an image generated by his AI system DABUS. However, District Judge Beryl Howell upheld the U.S. Copyright Office's rejection of the application, stating that human authorship is a fundamental requirement for copyright protection. Thaler plans to appeal the ruling, arguing that allowing AI copyrights would promote progress in science and the arts as outlined in the Constitution. But the court found centuries of precedent require works to have human creators to obtain copyrights. The case highlights challenging questions as artists increasingly utilize AI in their creative process, pushing the boundaries of intellectual property law.

§2. Advancement in Science

4/ How Global Scientists debunked the super-hyped superconductor candidate LK-99

In late July 2022, a team in South Korea published a preprint claiming LK-99, a compound of copper, lead, phosphorus and oxygen, was a superconductor at room temperature. This extraordinary claim quickly garnered attention as previous superconductors only worked at extremely low temperatures. Initial replication attempts by scientists did not observe room temperature superconductivity but were inconclusive. Over the next few weeks, mounting evidence disproved the claim:

• A team in China found LK-99's "levitation" was caused by ferromagnetism, not superconductivity's Meissner effect.

• Another Chinese team showed impurities of copper sulfide, not intrinsic LK-99, caused the sharp drop in resistivity initially claimed as proof of superconductivity.

• Calculations by a US-European team showed LK-99's electronic structure was incompatible with superconductivity.

• In mid-August, a German team synthesized pure LK-99 crystals, proving the material is an insulator without superconductivity.

Through rigorous experiments and analysis of samples, theory, and prior literature, the physics community conclusively demonstrated LK-99 does not superconduct at room temperature. The saga shows the importance of skepticism, precision, and global collaboration in reproducing extraordinary scientific claims.

5/ New treatments for Alzheimer’s offer hope—but raise questions, too

A new study shows donanemab, an Eli Lilly drug targeting Alzheimer's, significantly slowed disease progression. It marks steady progress, but headlines calling it a "breakthrough" are overblown. The costly drug has dangers and works only early on.

6/ A UK Startup claims to recycle heat from cloud computing

Heata, an English startup, has created an innovative cloud computing network that provides free hot water to people's homes by utilizing the heat generated from computer processors. Heata places servers in subscribers' houses that connect via WiFi to form a distributed cloud network which processes data for paying corporate clients. The heat emitted from the busy servers is captured by a conductor and transferred to a boiler which heats water for the home. Each server prevents 1 ton of carbon emissions annually and saves households about $250 on hot water costs. With data centers producing massive heat as AI workloads grow, Heata's model offers a sustainable way to repurpose computing byproduct for social good, while expanding access to an essential resource. The company has installed over 100 units in England to date, helping alleviate energy poverty and reducing cloud computing's environmental impact through an ingenious circular system.

§3. Developments in AI for Science

7/ GM invests big bucks in AI+Battery Design startup

General Motors has invested in the battery startup Mitra Chem to help accelerate the development and commercialization of new EV battery chemistries like lithium manganese iron phosphate (LMFP). Mitra Chem uses AI and machine learning to rapidly develop and test new battery formulations. The $60 million investment will support Mitra Chem scaling up operations and aim to cut the typical lab-to-production timeline by over 90%. GM hopes Mitra Chem's technology and expertise will reinforce its own battery R&D efforts, leading to more affordable and better performing cells. This aligns with GM's strategy to build a US-focused battery supply chain and transition fully to EVs across its lineup. Overall, the partnership aims to bring advanced battery chemistries like LMFP to market much faster to benefit GM's future EVs.

8/ AI-based drug discovery startup lands $200M

Genesis Therapeutics, an AI-focused biotech startup, has raised $200 million in Series B funding. The round was co-led by Andreessen Horowitz and an anonymous life science investor. The company uses AI to design novel small molecule drugs. It aims to speed up drug discovery and improve success rates.

The funding will support advancing Genesis' AI technologies, expanding its pipeline of potential drugs, and moving its first AI-designed drugs into clinical trials. Genesis has partnerships with major pharmas like Eli Lilly and Genentech.

9/ Pioneer of AI for Science awarded ICIAM Maxwell Prize 2023

The 2023 ICIAM Maxwell Prize has been awarded to mathematician Weinan E. The Prize - created with the joint support of the James Clerk Maxwell Foundation and the Institute of Mathematics and its Applications - is one of the world's top awards in applied mathematics, aimed at recognising originality in the field. The award is made every four years, to coincide with the four-yearly congress of ICIAM - the International Council for Industrial and Applied Mathematics - which this year will be in Tokyo in August.

Professor E's work has had an impact in areas including fluid dynamics, chemistry, material sciences, and soft condensed matter physics. He has carried out pioneering work in the application of machine learning, including the application of deep learning techniques to scientific computing.

He says that the advent of machine learning is transforming applied mathematics from a mass of techniques to a unified discipline. "In the history of science, there were two periods of time that made the most impact for applied mathematics. The first was the time of Newton, during which it was established that mathematics should be the language of science. The second was the time of von Neumann, during which it was proposed that numerical algorithms should be the main bridge between mathematics and science. Now the third time is at the horizon, a time when all the major components of applied math are in place, to form the foundation of not only interdisciplinary scientific research but also exciting technological innovation. This is truly an exciting time."

One more thing …

10/ Neuroscientists Re-create Pink Floyd Song from Listeners’ Brain Activity

Neuroscientists have for the first time reconstructed a song - Pink Floyd's "Another Brick in the Wall, Part 1" - from brain activity data recorded while people listened to the track. Using electrodes placed on listeners' brains and machine learning algorithms, the researchers captured the electrical signals generated in response to musical elements like tone, rhythm, and lyrics. These signals were then decoded by an AI model and processed to recreate an approximate rendition of the original song, with garbled but recognizable lyrics. While still rudimentary, the technique demonstrates the potential to one day use brain-computer interfaces to help people who have lost the ability to speak regain their voice and communicate more naturally.

Disclaimer: This page contains links and summaries of articles that have been curated from various sources. The curator makes no claims as to the accuracy, completeness, or reliability of the information provided in the original articles. Links are provided for reference and convenience only. Readers should evaluate the source and credibility of any content referenced in this newsletter and proceed at their own discretion.

Artificial intelligence, or AI, has rapidly transformed various facets of our lives, from voice recognition and image processing to forecasting financial markets and making medical diagnoses. But in the complex world of quantum physics, the leap forward has just started.

Scientists have been deploying AI to unlock the complexities of the atomic world for quite some time now. They have been particularly interested in modeling the potential energy surface (PES) - a vital concept that represents the energy of a system as a function of the positions of all its atoms. The PES is the Rosetta Stone of atomic systems, holding the key to understanding the system's properties and behaviors.

Now, researchers have put forward a Deep Potential model with a novel attention mechanism, called DPA-1, which shows promise in learning and interpreting the PES. It's the interpretability part of this equation that recently gave us an unexpected surprise.

The researchers trained the model on various atomic systems but left out one particular element - Lithium (Li). In a captivating twist, they decided to use the AI model to "re-discover" lithium, an element sitting between Hydrogen (H) and Sodium (Na) on the periodic table.

To do this, the researchers generated a new representation for lithium by blending the AI's learned representations of hydrogen and sodium. They then adjusted the blend until the model's performance significantly improved. Astonishingly, this occurred when the representation was 70% sodium and 30% hydrogen, in line with the atomic properties of lithium in relation to sodium and hydrogen.

This "re-discovery" of lithium is significant for several reasons. For one, it illustrates the model's ability to transfer what it has learned about certain elements to others it has not encountered before - a trait known as transferability. Secondly, it provides insights into the internal workings of the AI model, showing that the learned representations are not arbitrary but correspond with the structure of the periodic table - a characteristic known as interpretability.

Finally, it offers a proof of concept for the use of AI to pretrain on a broad dataset and then fine-tune on specific tasks, a strategy that has become increasingly common in machine learning applications. This combination of transferability, interpretability, and pretraining and fine-tuning capabilities is like hitting a trifecta in the AI race.

Is this a whimsical quirk of serendipity or a significant breakthrough? While the experiment is captivating, it's just one piece of the puzzle. The real test lies in the model's ability to generalize these findings to other unseen elements and across different types of tasks and datasets. And while this AI experiment doesn't advance our understanding of quantum physics per se, it certainly illuminates how we can leverage AI models to learn, represent, and apply domain-specific knowledge in powerful ways.

Like the most inspiring scientific journeys, our re-discovery of lithium through AI is just a beginning. The spiral of innovation continues, and as we delve deeper into the atomic realm, the true potential of AI in quantum physics is yet to be discovered.

AI for Science’s Pursuit of Dimension-independence

Many challenges in scientific computing and applied mathematics involve dealing with high-dimensional problems. This high dimensionality often arises from multiscale effects. For example, the dimensionality of quantum many-body systems grows exponentially with the number of electrons included. And microscopic scale models in multiscale modeling often have a very large number of degrees of freedom. The difficulties caused by excessive dimensions are also known as the “curse of dimensionality”.

Traditional smoothness regularization methods clearly fail in high dimensions, unable to effectively describe the complexity of functions. Recent research has shown that approximating functions with specific neural network architectures provides a better measure of complexity. This has led to new function spaces associated with particular machine learning models, such as RKHS and Barron spaces, laying new foundations for analyzing high-dimensional function approximation. Meanwhile, machine learning models themselves have demonstrated powerful capabilities in approximating high-dimensional functions, enabling solutions to many previously intractable control theory and PDE problems due to the curse of dimensionality.

Various AI algorithms, including reinforcement learning and transfer learning, are not only expanding the scope of AI4Science models through continuous optimization and improvement, but also reducing model training costs. In the computer age, we often solve real-world problems by mapping physical models onto computers for mathematical simulation. However, more microscopic models, although more accurate as they are closer to first principles, face more severe “curse of dimensionality” due to their greater complexity and larger number of degrees of freedom. In contrast, more macroscopic models are simpler, more efficient, but less accurate. In this case, we need to find a way to reduce computational complexity while maintaining accuracy.

In statistical physics, the Monte Carlo method is a solution to this “curse of dimensionality” (see equation below). Its convergence efficiency in sampling depends on the variance of the function and the number of samples, but not the input dimension, thus making it dimension-independent.

In the AI4Science era, neural network functions (see equation below) have properties similar to Monte Carlo methods. For function spaces we care about, approximation errors depend on intrinsic variances related to the function rather than the dimensionality of the function.

Leveraging this trait of AI algorithms, we can use them as bridges between multiscale physical models, effectively approximating solutions of microscopic models. This allows microscopic models to become data generators, then through learning under AI, fuse corresponding models into more macroscopic ones. Through such iterative interactions, we can find a solution balancing microscopic accuracy and macroscopic efficiency. Using neural networks to approximate the policy function in dynamic programming can solve stochastic control problems with hundreds or even higher dimensions. Solving high-dimensional Hamilton-Jacobi-Bellman equations also becomes promising[1].

In practice, how to build bridges between AI and science is the core innovation point. Currently, the approaches differ for various scientific scenarios. In control problems, the goal is to find a policy function that optimizes an objective function (usually integral of state trajectories). Traditional dynamic programming methods need to solve a Bellman equation with dimension equaling the state space dimension, thus facing the curse of dimensionality. To address this, researchers propose representing the policy function with neural networks, viewing the control objective as a loss function, and the control dynamics as constructing a deep residual network. The network parameters can then be trained using stochastic gradient descent. Such deep learning control algorithms can already handle stochastic control problems with over a hundred or even higher dimensions. The Hamilton-Jacobi-Bellman (HJB) equation also plays a core role in control theory. In recent years, progress has been made in solving high-dimensional HJB equations using adaptive deep learning algorithms. This enables better handling of complex control problems in continuous state spaces.

In multiscale modeling, microscopic models contain too many details to be directly used for practical simulations. And traditional uniform spatial discretization algorithms also struggle to handle such high-dimensional microscopic models. Machine learning methods can identify macroscopic or collective variables from microscopic simulations, establishing connections between different scales and enabling direct multiscale coupling. Recent research shows this approach can extend atomic-level simulations to systems containing over a billion atoms.

For building physical models from data, machine learning also provides new ideas. Merely fitting data is not enough. Physical constraints need to be combined and training should use representative data, so that the model is interpretable and guarantees extrapolability. This brings hope to many problems that are hard to model starting from general physical principles. As long as physical constraints are satisfied, such models can achieve reliability comparable to general physical models.

3 Levels of “Enforcing” Physical Constraints onto Machine Learning’s “Black box”?

• Classical AI: From the perspective of "classical AI", improving a model's physical fidelity mainly comes from enhancing the quality of its training data. This is the weakest form, simply embedding physical knowledge into the training data itself. For example, generating data satisfying conservation laws or known symmetries for training. Such end-to-end utilization of data for modeling is fast and easy, but cannot guarantee the model will obey physical laws during inference.

• “Physics-informed”: A soft form is indirectly strengthening physical laws through the loss function. For example, Physics-informed Neural Network (PINN) incorporates differential equations into part of the loss function, guiding parameter optimization. PINNs embed differential equations and other physical constraints into the neural network structure to approximate functions containing physical laws. The inputs to a PINN include independent variables x and parameters λ, and the output is the solution u(x,λ). Network parameters are trained by minimizing a loss function consisting of two parts: 1) Data fitting - using network output u(x,λ) to approximate given data; 2) Physical constraints - taking derivatives of u(x,λ) and substituting into differential equations to require satisfying the equations. This is implemented via automatic differentiation. After training, the lowered loss function means an u(x,λ) satisfying both data and physical constraints is found. Variational methods also belong here, optimizing a functional (such as action in classical mechanics or free energy in statistical mechanics). Such forms can partially constrain the model, but will not rigidly require obeying physical principles.

• “Physics-enforced”: A stronger form directly builds fundamental physical laws into the model architecture. Using descriptors preserving problem symmetries is a typical example. Another is constructing Hamiltonian or Lagrangian neural networks with structures respecting certain physical conservation laws like energy. This form maximally ensures model behaviors conform to physical principles, but also restricts model expression. In DeePMD, researchers cleverly incorporated some key physical constraints: 1) Translation invariance - the total energy of a physical system does not depend on absolute atomic positions. This is achieved by expressing the system energy as a sum over atomic energies, each determined by the atom's local environment defined relative to the atom, thus ensuring translational invariance. 2) Rotation invariance - the total energy should not change under system rotations. DeePMD implements this by using rotationally invariant local descriptors. 3) Permutation invariance - swapping atom indices should not change the total energy. DeePMD also guarantees this through summing atomic energies. These invariances are key components of the DeePMD design. The model is trained on potential energy surfaces from quantum mechanics calculations. The invariances help ensure the learned model can generalize well to new systems. Without them, the model would struggle to predict properties of systems different from the training data.

It should be emphasized that stronger physical constraints are NOT “better” by default, but depend on the specific factors for each case (such as data quality, and the end user’s actual expectation). For example, AlphaFold does not impliment very strong constrains, yet thanks to the extremely high quality of the PDB data source, its training is very successful, with excellent inference performance and efficiency. Meanwhile, introducing strong L2 constraints sometimes decreases model trainability, impacting time and cost. When constructing AI for Science algorithms for specific scenarios, pragmatism is always a good idea.

Advances in AI for Science depend not only on applying AI algorithms, but also on improving and enhancing many classical algorithms. With the lowering of barriers to acquiring high-quality and large-scale data, how to better fuse data from different sources, scales, and modalities has become an important challenge. For example, in weather forecasting, the spatiotemporal scales corresponding to different observed indicators have large differences. How to utilize such data with differences to enable more accurate and efficient forecasting poses real algorithmic challenges, requiring us to improve classical data fusion algorithms to adapt to new demands. The tremendous imagination space for AI for Science lies in how to better utilize AI algorithms to connect scientific computing and physical models, guiding scientific and industrial innovation. The power of AI lies in its great potential to solve complex problems, thus advancing scientific research and technological development.

Today, the bottleneck in research is not only "how to solve problems", but also "how to define problems" and "how to choose tools". For example, when solving PDEs, if the main difficulty is high dimensionality, PINN would be very effective. But if the challenges do not lie in the curse of dimensionality, but rather geometrical complexity, multiscale effects, etc., the advantages of neural networks would not easily emerge. Instead, difficulties like nonlinear optimization would lead to low solving efficiency, uncontrollable errors, and difficulty improving systematically. In such scenarios, choosing the Random Feature Method (RFM) could avoid mesh generation and easily handle complex geometries.

Deep understanding of the problem is the first step to solving it. The innovations in AI for Science algorithms come not only from the ever-changing AI models, but even more so from scientists' analysis, diagnosis and “translation” of specific scientific challenges, in order to maximize the effectiveness of AI in the scientific domain.

Source:

[1] Weinan E, "The dawning of a new era in applied mathematics" , Notice of the American Mathematical Society, April, 2021.

[2] Jingrun Chen, Xurong Chi, Weinan E & Zhouwang Yang. (2022). Bridging Traditional and Machine Learning-Based Algorithms for Solving PDEs: The Random Feature Method. Journal of Machine Learning. 1 (3). 268-298. doi:10.4208/jml.220726

You never know when or where AI and Science would converge

§1. Machine Learning and Intelligent Systems

1/ 1Yr → $4Bn?! AIGC Funding Space Remains Red-hot & Controversial

AI startup Inflection, led by former DeepMind executive Mustafa Suleyman, raised $1.3 billion in new funding just months after launching chatbot Pi. The round, led by Microsoft, Nvidia, and billionaires like Reid Hoffman and Bill Gates, values Inflection at $4 billion. How much of the round is cash vs. compute credit is unknown and has raised controversy.

Inflection claims it now has the world's most powerful AI computing infrastructure with over 20,000 Nvidia GPUs, aided by partnerships with Microsoft and Nvidia. Suleyman says Inflection will keep expanding its capabilities to develop bigger AI models, raising billions more despite critics arguing such large models are unnecessary, harmful, and environmentally damaging. Inflection believes massive scale is required to lead the "tidal wave" of AI, but its voracious growth and funding has sparked debate on AI funding.

2/ Berkeley-based startup raises $200 million from Bill Gates etc. to dig for copper, lithium using AI

Berkeley-based KoBold Metals, which leverages artificial intelligence to guide mineral exploration, raised $200 million in a funding round valuing the company at over $1 billion. The round was led by T. Rowe Price and included Bill Gates' Breakthrough Energy Ventures, Andreessen Horowitz, and others. KoBold combines clean energy and AI, two hot investment areas, to improve the efficiency of discovering critical battery metals like copper and lithium. It uses machine learning to analyze geological data and identify high-potential spots to explore. The funds will accelerate efforts to secure materials needed for electrification, as demand outpaces supply. While skeptical at first, mining giants like BHP now partner with KoBold, recognizing that remaining deposits are deeper underground and harder to find with traditional techniques. KoBold aims to transform mineral exploration with a scientific, tech-driven approach to mining.

3/ AI-copyright debate heats up

A growing group of artists, writers, and filmmakers argue AI chatbots like ChatGPT and image generators were illegally trained on their copyrighted work without consent or compensation. Recent lawsuits against AI companies allege violations of copyright law. While the tech firms claim fair use protections, critics say scraping creative content to train AI models is unethical and harms artists' livelihoods. Lawmakers, social media platforms, and entertainment unions are siding with creators demanding more transparency and control over use of their content in AI training data. Possible remedies include licenses, filters to prevent copying, or curated datasets. The battle poses a major threat to adoption of AI systems that tech giants argue are an innovation on par with the mobile phone. Resolution may require changing how AI data collection works to address rights and consent issues.

4/ China became first major country to regulate generative AI, and the rest of world are taking notice

China announced comprehensive new rules governing generative AI like ChatGPT, to be overseen by its top internet regulator. The regulations require adherence to social values and ban illegal uses. Developers must register algorithms and undergo security reviews, extending existing controls. Exempted are research AI and systems for foreign use, hinting at global ambitions. The rules offer a potential model for AI governance, including intellectual property protections and privacy safeguards. They arrive amid an AI race with the US, as China sees the technology as an economic priority. With encouragement for global rule-setting participation, China may advocate its regulatory approach internationally. The broad regulations demonstrate the country's intent to rapidly adopt AI across industries, but also firmly embed government oversight. China's assertive stance makes AIGC regulation an emerging frontier in great power technological competition.

§2. Advancement in Science

5/ Superconductor Research Remains Heated

The recent discovery of pressure-induced superconductivity above 80K in the nickelate La3Ni2O7, as reported in Nature on July 12th, 2023, represents a major breakthrough for high temperature superconductivity research. After 36 years, scientists seem to have finally found a second family of unconventional superconductors beyond copper oxides that can exceed the liquid nitrogen temperature of 77K, rekindling hopes of unraveling the mechanism and enabling applications of high temperature superconductivity. Despite the many obstacles faced, including theoretical challenges and practical limitations of existing materials, persistent explorations by scientists worldwide have uncovered new superconducting systems like iron-based compounds previously. The latest finding in nickelates, enabled by high pressure techniques, offers a new platform to investigate unconventional superconductivity. Further discoveries in this system may lead to higher transition temperatures or properties better suited for applications. The eternal allure of superconductivity continues to drive human ingenuity in materials science and condensed matter physics.

§3. Developments in AI for Science

6/ AI yields designer proteins

World renowed computational biologist David Baker’s team published a new AI program RFdiffusion, which allows researchers to generate novel proteins tailored for specific functions, similar to how Midjourey/DALL-E/StableDiffusion produces images from text prompts. Scientists can upload a 3D model of a biological molecule like a cell receptor. RFdiffusion can then predict protein sequences and structures likely to interact with the target, providing promising candidates to test as therapies or for other applications. In evaluations, RFdiffusion increased accuracy up to 100 times compared to previous protein design programs not using AI. This new AI-powered approach could assist researchers in designing proteins for goals such as extracting metals from seawater or removing carbon dioxide from the air. By leveraging AI to predict optimal protein structures, RFdiffusion aims to accelerate discovery of proteins useful for medicine, environmental remediation, and more.

7/ World Economic Forum Taking Notice of AI for Science

Here are the key points by Joris Poort, CEO of Rescale, writing on WEF’s platform:

• AI is transforming fields like engineering and scientific computing to redefine innovation across industries. Computer simulations increasingly use AI to understand how different product designs will perform.

• AI can make R&D more efficient by reducing the need for expensive and time-consuming physical simulations. Machine learning models can be trained on simulation data to quickly predict how new designs will perform.

• AI-physics models capture engineering best practices and expertise that can be retained within an organization. This brings greater agility to develop new prototypes and enter new markets faster.

• Generative design AI tools can come up with innovative product designs that engineers may not have conceived of on their own. This can lead to new and unconventional product shapes and forms.

• Organizations need to establish best practices to ensure they navigate the AI transition safely. This includes reviewing AI outputs for accuracy and ethics, securing IP, and providing the right training data.

• With appropriate safeguards, AI can accelerate engineering innovation, scientific discovery, and the development of breakthrough products and designs. Supporting AI physics will help create better products faster.

8/ AI-designed drug enters human trials

Biotech company Insilico Medicine has started a phase 2 human trial testing a drug for idiopathic pulmonary fibrosis that was discovered and designed entirely by artificial intelligence. This marks a significant milestone as one of the first mid-stage trials of an AI-generated therapy. Insilico founder Alex Zhavoronkov claims the company's AI platforms can dramatically accelerate drug discovery and slash development costs. The generative AI rapidly identifies novel targets and designs molecule compounds. While AI's potential in drug development is still developing, pharma companies have invested billions in partnerships with AI startups like Insilico seeking to capitalize on potential cost and time savings. Critics caution AI's promise may be overhyped. But success for Insilico's self-developed drug candidate could pave the way for AI end-to-end across the pharma R&D process.

9/ New AI Weather Prediction Published on Nature

Two new AI models detailed in Nature could significantly improve weather forecasting speed and extreme weather predictions. Huawei's Pangu-Weather uses deep learning to analyze all weather variables simultaneously, generating forecasts in seconds versus hours for traditional methods, with comparable accuracy. Pangu-Weather accurately predicted tropical cyclone paths despite no training data on them, showing AI's ability to generalize physics. Separately, NowcastNet leverages physics-based generative AI to forecast extreme rainfall further in advance than other leading systems. While questions remain on real-world performance, AI offers major potential to complement conventional forecasting and provide more advance warning for extreme weather events. However, AI may underestimate event intensity, and climate change poses challenges. Overall, AI looks set to transform meteorology, but human expertise will still be critical.

One more thing …

10/ Biotech Company’s Stock Surge 78% on Nvidia’s $50M Investment

We covered Recursion on the June Issue of Intelli-Science Monthly when it acquired two smaller companies in the AI Drug Discovery space. Recently Nvidia maked a PIPE investment in Recursion, which prompted its stock to surge over 78% in one day. AI for (Life) Science is a multi-layered industry with players on hardware, fundemental model, specialised model, data, data infra, application etc. As the industry develops during its infant stage, collaborations and consolidations activities will likely to remain vibrant.

You never know when or where AI and Science would converge

§1. Machine Learning and Intelligent Systems

1/ AMD bid for a seat in the heated AI chip war.

AMD accouned that the coming MI300x will feature 192GB memory which is capable of hosting the 40BN parameter Large Language Model Falcon.

2/ Growth stage SaaS/AI startup Dialpad, accounced that it reaches $200M ARR

The company, which started in call center softwares, credits its growth largely to its AI products. The company acquired TalkIQ, an early adopter of AI in language processing (ex. call centre quality control), back in 2018.

3/ Experiment showed AI struggled to solve murder-mystery puzzle book.

The 1934 book "Cain’s Jawbone" are intentionally shuffled to test AI’s ability to reorder. The book was digitized and given to data scientists for the 2022 Cain’s Jawbone Murder Mystery Competition, using natural language processing algorithms to reorder the pages. But the stylized, vague language and false clues hindered progress. Even the best AI only correctly ordered 42% of the book, showing AI's limitations without context.

4/ Private Equity Giants join AI goldrush

KKR is acquiring CoolIT Systems, a company specializing in data center cooling systems, for $270M. This move aims to meet the rising demand from data-center operators, including major cloud providers, amid the surge in generative AI and other power-demanding applications. Data center is definitely not the “cool” part of the current AI goldrush, but by no mean is it trivial.

§2. Advancement in Science

5/ Superconductor Controversy Continues

Earlier in March, the controversial scientist Ranga Dias published a paper on Nature claiming that he has achieved meaningful room-temperature superconductivity with a novo material (which he has refused to specify, citing business confidentiality). Room temperature superconductivity is considered to be a pre-requisite for nulear fusion technology, and therefore a pre-requisite for unlimited clean energy. So a meaningful breakthrough in this area is of immense importance for the entire human civilisation.

However, almost immediately after Dias’s paper, scientists around the world sought to repeat his result with little progress. Most noticably, Dr. Hai-hu Wen of Nanjing University also published a paper on Nature, which seems to directly disprove Dias’ result.

Another turn occured in June, when renowned scientist Russell Hemley at UIC released his result, which seems to verify Dias’s work. It seems the acedemic world would still need some time to reach consensus.

6/ Can we see protein in motion?

Two independent research teams used a super-resolution tool, MINFLUX, to study kinesin - molecular motors powering cell processes - in near-real time under physiological conditions. The studies revealed previously unseen details of kinesin motion, including speed, stride length, and movement patterns within living cells. This advancement offers insights into molecular behavior and has potential implications for drug development.

§3. Developments in AI for Science

7/ Can AI discovery new science? YES

In the summer of 2023, prominent AI for Science pioneer DeepMind published an article on Nature, demonstrating that its AI agent, AlphaDev, used reinforcement learning to discover enhanced sorting algorithms – surpassing those honed by scientists and engineers over decades. Sorting is used by billions of people every day without them realising it. It underpins everything from ranking online search results and social posts to how data is processed on computers and phones, so an improvement at this level is of system significance.

As if people's minds are not blown enough, a mere few days later, Dimitris Papailiopoulos, associate professor at University of Wisconsin-Madison, announced on Twitter that he has successfully prompted GPT-4 to discover the same breakthrough Alphadev did. This stirred up a frenzy on the social media platform, eventually catching the attention of its eccentric billionaire owner, Elon Musk. The fact that two different AI were able to discover this new "science" makes it even more exciting as it demonstrates, perhaps the first time since the Enlightenment, a scalable path towards scientific discoveries.

8/ Sanofi “all-in” AI for Life Science

Sanofi rolls out its AI-powered app, plai, developed with Aily Labs, providing 360° insights across all company activities for data-driven decisions. The app accelerates processes from R&D to manufacturing, reducing research times and enhancing clinical trial design. Sanofi aims to become the first pharma company fully powered by AI, transforming the practice of medicine.

9/ Consolidation in AI for Life Science

Recursion (NASDAQ: RXRX), a leading clinical stage TechBio company decoding biology to industrialize drug discovery, today announced it has signed agreements to acquire two companies in the AI-enabled drug discovery space: Cyclica and Valence. Recursion has entered into agreements to acquire Cyclica for a purchase price of $40 million and Valence for a purchase price of $47.5 million

First Take …

10/ Overshaddowed by Vision Pro, the M2 Ultra might be more exciting than one would think

Though Apple's product launch didn't emphasize its AI capabilities, Mac power users like myself have long awaited such a product. In the wave of AIGC, the Mac series had a mediocre AIGC experience due to lack of software/hardware support. However, with the M2 Ultra, there's buzz about it running Falcon 40B, a large language model with 40 billion parameters. Theoretically, the M2 Ultra's 192GB memory should suffice, and the feasibility of M2 running LLaMa 13B has been confirmed.

Remembering the releases of the first iPhone, touchID, faceID, and vision pro, Apple doesn't chase tech premieres but aims for a clean sweep. Regardless, we shouldn't think that one of the world's top chip design companies would stay passive during this AI wave.

June 15, 2023 – In a recent cover article, "Uni-Dock: GPU-Accelerated Docking Enables Ultralarge Virtual Screening," published in the Journal of Chemical Theory and Computation, DP Technology has introduced Uni-Dock, a GPU-accelerated high-performance molecular docking engine.

This technology allows an acceleration of molecular docking calculations up to 1,600 times faster than a single-core CPU on an NVIDIA V100 GPU, while preserving computational accuracy.

Leveraging Uni-Dock, the research team successfully completed a multistage virtual screening of 38.2 million compounds from the Enamine Diverse REAL drug database on the KRAS G12D target within just 11.3 hours, using a cluster of 100 NVIDIA V100 GPUs. The screening's average speed exceeded 37,000 molecular docking computations per GPU per hour, which substantially reduces the time and cost needed for ultra-large scale virtual screenings, thereby enabling efficient exploration of extensive chemical spaces during the early stages of new drug development.

Methodology

Traditional molecular docking follows a set process: A global search of protein-ligand complex conformation is conducted using the Monte Carlo (MC) method, factoring in ligand rotatable bond dihedral angles and ligand positions. The energy and force of the ligand in the current complex conformation is calculated, followed by local optimization using the BFGS algorithm to obtain the locally most stable conformation of the complex. This process is repeated until all search steps are exhausted, and the protein-ligand complex conformation with the lowest energy is returned.

Uni-Dock effectively accelerates the docking of single molecules by simultaneously launching multiple conformation search threads for a single ligand in a GPU (Kernel 2 in the graph below), reducing the computational volume per search thread by decreasing the MC iteration number for each thread. It also simultaneously launches multiple ligand docking computations to make full use of the GPU's computational capabilities (Kernel 1), dynamically assigns parallel ligand numbers based on GPU memory space, and aims to maximize ligand throughput to offset the additional cost of launching computational kernels.

Through computational logic optimization, some calculations generating massive information are transferred to the GPU, reducing host-device data transfer. Uni-Dock converts some calculations with lower precision requirements to single-precision calculations to accelerate computation and reduce GPU memory usage. Asynchronous mechanisms balance the time for CPU file reading/writing and GPU computation, and GPU memory is intelligently scheduled and dynamically allocated to fully utilize GPU computational performance across various GPU models.

Performance over Incumbant Softwares

With comparable precision to AutoDock Vina (exhaustiveness=32), Uni-Dock achieves speed enhancements of 120x, 925x, and 1627x in three optimization stages. The team compared Uni-Dock with two other GPU-accelerated molecular docking software, Autodock-GPU and Vina-GPU, in terms of docking efficiency and accuracy, defining three computational complexity levels as Fast Mode, Balanced Mode, and Detailed Mode. In terms of efficiency, Uni-Dock was over 10 times faster than both competitors, with docking speeds of approximately 0.10s/ligand, 0.32s/ligand, and 0.38s/ligand across the three modes. In terms of accuracy, Uni-Dock and Vina-GPU, both based on AutoDock Vina, had comparable precision,

DP Tech's research and development team set a new performance benchmark for the latest Uni-Dock high-performance molecular docking engine, maintaining comparable accuracy across all optimization stages to the existing AutoDock Vina 1.2 engine. Using eight targets in the DUD-E database, the performance was measured based on the enrichment capability of AutoDock Vina 1.2 (with the settings: exhaustiveness=32, Vina scoring function, semi-flexible docking). The results showed that Uni-Dock achieved an impressive 1627-fold acceleration over AutoDock Vina when using an NVIDIA V100 32G GPU, compared to a single Intel® Xeon® Platinum 8269CY (Cascade Lake) 2.5 GHz CPU core.

To validate Uni-Dock's multidimensional accuracy, the team compared the screening and docking power of Uni-Dock and AutoDock Vina across 102 protein targets in the DUD-E dataset (categorized into eight types) and 285 protein-ligand complexes in the CASF-2016 dataset. This comparison reaffirmed that Uni-Dock consistently matched the computational precision of AutoDock Vina at every level.

Additionally, Uni-Dock outperformed two other GPU-accelerated molecular docking software, Autodock-GPU and Vina-GPU, in both efficiency and accuracy. For an equitable comparison, three computational complexity levels were defined as Fast Mode, Balanced Mode, and Detailed Mode. In all categories, Uni-Dock exceeded its competitors by over tenfold, with docking speeds of approximately 0.10s/ligand, 0.32s/ligand, and 0.38s/ligand respectively.

Further tests demonstrated Uni-Dock's outstanding ability to linearly scale with the number of GPUs and effectively adapt to GPUs of different models and architectures. This ensures efficient deployment on varied computational resources and optimal utilization of large-scale clusters for high-throughput virtual screening.

Case Studies

DP Tech's team conducted a virtual screening on the KRAS G12D target using the Enamine Diverse REAL drug database with Uni-Dock. The database, consisting of about 38.2 million molecules, was screened using a stratified approach to balance speed and accuracy. The screening took just over 11 hours with an average rate of more than 37,000 molecular dockings per GPU hour, highlighting Uni-Dock's efficient screening capabilities.

DP Tech's Uni-Dock engine fully leverages the power of GPU parallel computing and memory space to achieve over 1600 times the acceleration ratio of AutoDock Vina, with comparable precision. It outpaces other GPU-accelerated molecular docking engines by over tenfold, achieving a molecular docking efficiency of 0.1s/ligand, and can complete virtual screening of more than 38.2 million molecular databases in less than 12 hours.

Available for FREE on Cloud-native Bohrium® Notebook and more

DP Technology has announced that Uni-Dock is now freely available to academic users! Following the usage agreement, researchers can obtain the latest release of Uni-Dock from DP Technology's GitHub repository here and use the accelerated capabilities of Uni-Dock to propel their scientific research forward.

If you're a beginner eager to learn how to use Uni-Dock, from mass submission of screening tasks to result recovery analysis, step-by-step instructions are available on the Bohrium® Notebook. Uni-Dock use-cases can be accessed here and guide users through the process.

If you'd like to conduct a virtual screening task with Uni-Dock but do not have access to a GPU machine, DP Technology also offers its Launching platform to external users. Simply submit your virtual screening job here, and the Launching platform will automatically allocate computing resources to your task, swiftly completing the virtual screening and delivering the results.

In addition, if you wish to visualize and analyze the binding poses of protein-ligand complexes, conduct further evaluation and analysis tasks such as protein structure prediction, MM-GB/PBSA, free energy perturbation (FEP) calculations, molecular property prediction, and antibody humanization design and property prediction, DP Technology offers the Hermite® drug computational design platform. Available at hermite.dp.tech, Hermite® provides a one-stop drug design solution.

Hermite® brings a novel, web-based, interactive molecular display experience for drug development scientists. It facilitates cross-window intelligent collaboration and offers diverse molecule display and operation features, making it easier for users to view, analyze, and share protein, drug molecules, and their simulation data. Hermite® supports both local and cloud-based private deployments, further enhancing its versatility and usability.

About DP Technology

DP Technology, Ltd. is an AI research and development company aspiring to solve the biggest problems in science. DP Technology is founded on the fact that our current scientific tools and computing methods have not been adequate in deepening our understanding of the microscopic world. Specifically, they have failed in predicting the behavior of complex systems at multi-scale levels. This has resulted in prolonged stagnation in the development of new materials, new drugs, new semiconductor frameworks, and beyond. We believe artificial intelligence has the potential of solving these imminent global challenges and the best way to do it is by combining the best practices of AI, scientific computing, and cloud HPC, designing innovative algorithms and software, and connecting cutting-edge technologies with real challenges in industry.

Photo by Andrea De Santis on Unsplash

In the ceaseless quest for knowledge, humanity has always sought more effective ways to navigate the vast, labyrinthine landscape of understanding. Over time, our tools have evolved, from the primitive to the intricate, mirroring our own intellectual evolution. Today, standing on the brink of a new era, we find ourselves at an intersection where artificial intelligence collides and merges with the pursuit of science. At the center of this converging point, Large Language Models (LLMs) stand poised as transformative agents, promising to reconfigure our interactions with the vast domain of human knowledge.

As ChatGPT rose to prominence, many tailored tools have been created by the community in adopting GPT and other LLMs for scientific research tasks. Here, we would like to systematically explores the transformative potential and challenges of LLMs in the context of scientific research. We begin by unveiling the concept of LLMs as a two-way interface for human knowledge, a dynamic medium that fosters a symbiotic relationship between the human intellect and the encyclopedic domain of scientific literature. By acting as both a distiller and facilitator of knowledge, LLMs hold the potential to drastically reduce barriers to entry for individuals delving into new scientific fields, while simultaneously refining the process of knowledge contribution.

Next, we delve into the need for continuously evaluating and amplifying LLMs' proficiency in dealing with scientific queries. Drawing parallels to the pioneering works of intellectual giants like Hilbert, Godel, and Turing, we explore the intriguing intersection of formal proofs and natural language. Here, we uncover the powerful, albeit complex, potential of LLMs to grapple with scientific concepts, pushing us to relentlessly refine their ability to comprehend, interpret, and eventually contribute to mankind's domain of scientific knowledge.

In the final section, we journey into the realm of LLMs' "Chain of Thoughts," exploring their capacity to simulate the scientific method and mimic a scientist's approach to problem-solving. Despite their lack of consciousness, we posit that LLMs' 'thought simulation' opens exciting avenues for hypothesis generation, experimental design, and critical review.

It is invigorating to postulate a future where LLMs become an integral part of scientific research. As we stand on the precipice of this new era, we are tasked with not only understanding these transformative potentials but also with guiding this symbiosis of artificial and human intelligence. In doing so, we contribute to shaping an exciting future where our collective ability to investigate, understand, and expand the frontiers of knowledge is unimaginably amplified.

I. LLMs as a Human-Knowledge Interface

Humanity's relentless pursuit of enhanced knowledge acquisition, storage, and dissemination methods has charted a course from primitive inscriptions to the sophisticated digital databases of today. In this relentless march of progress, Large Language Models (LLMs) like GPT-4 embody the vanguard, boasting the potential to revolutionize our interaction with scientific knowledge.

At the heart of this revolution lies the concept of LLM as a two-way interface, a dynamic medium through which humans can both extract and contribute knowledge. This two-fold functionality redefines the relationship between humans and the boundless ocean of scientific literature.

On one hand, LLMs enable the extraction and synthesis of knowledge in an efficient, accessible manner. By deciphering and presenting complex scientific information, they drastically lower the barriers of entry into new fields. This ensures that even newcomers, who might otherwise be deterred by the intricacies of specialist literature, can navigate and comprehend scientific discourse. ChatPDF, for example, allows users to upload a PDF file (usually a scientific paper) and then ask the AI agent questions about this paper, as if a student seeking clarification at a professor's office hour.

On the other hand, LLMs can accelerate and refine the process of knowledge contribution. For instance, harnessing an LLM's capacity for multi-step inference and decision-making, researchers can navigate the labyrinthine expanse of scientific literature to identify the most pertinent papers. This doesn't merely expedite the literature review process; it also enhances the scientific paper's quality by ensuring a thorough and relevant survey of existing knowledge.

Furthermore, LLMs can play an instrumental role in the composition of scientific papers. By assisting in language production, they can serve as indispensable aids to scholars, particularly those for whom English is a second language. More than just linguistic assistance, LLMs can help structure logical narratives and ensure coherency, a function that is pivotal in scientific discourse, where complex ideas and findings must be meticulously organized and presented.

By functioning as both a distiller of existing knowledge and a facilitator of new knowledge creation, LLMs could turbocharge scientific progress. This dual role, encapsulating knowledge extraction and contribution, envisions LLMs as more than just tools - they could be collaborative partners, poised to reshape the knowledge landscape in an unprecedented way.

II. Evaluating and Amplifying LLMs' Scientific Proficiency

Let us also consider the fundamental compatibility between LLMs and the nature of human scientific knowledge. Our understanding of the universe is encoded in language, both natural and mathematical. The essence of science lies in its narrative, in the hypotheses posited, the methodologies explained, the results shared, and the conclusions drawn—all through the medium of language. This narrative nature of scientific knowledge makes it a fitting task for LLMs to handle.

Reflecting on the works of eminent thinkers such as David Hilbert, Kurt Gödel, and Alan Turing, one might draw parallels between their philosophical and theoretical explorations and the functioning of LLMs. Hilbert's pursuit of a complete and consistent system resonates with our aspirations for LLMs—to create a model that can fully and accurately represent our scientific knowledge. In essence, these formal proofs function as structured, mathematical narratives which are not dissimilar to natural language. Given this perspective, LLMs, trained extensively on natural language data, should theoretically possess a certain aptitude for grappling with scientific concepts. Yet, as Godel's Incompleteness Theorems suggest, no system can be both consistent and complete. Even with the best of our tools, some questions will remain beyond reach. Turing’s work on undecidable problems reinforces this, while also suggesting a provocative counterpoint – that LLMs might help identify such boundaries in our knowledge systems.

LLMs, while already exhibiting an impressive ability to handle scientific queries, require continuous assessment and improvement. To truly unlock the potential of LLMs in science, we must continuously refine their ability to comprehend, interpret, and generate scientific content. This includes a nuanced understanding of not just natural language, but also the languages of mathematics and logic that underpin scientific discourse. Herein lies a formidable challenge, but also an exhilarating opportunity. As we augment the proficiency of LLMs in the domain of science, we are essentially turbocharging our collective ability to investigate, understand, and expand the frontiers of knowledge.

An exciting step taken in this direction is the integration of WolframAlpha into ChatGPT. WolframAlpha, which incorporates the vast knowledge of Mathematica, will undoubtedly expand ChatGPT's ability to tackle complex science questions. As we stand at the cusp of an era where LLMs become an integral part of scientific research, it is imperative that we understand the transformative potential they hold.

III. One more thing ...

In the summer of 2023, prominent AI for Science pioneer DeepMind published an article on Nature, demonstrating that its AI agent, AlphaDev, used reinforcement learning to discover enhanced sorting algorithms – surpassing those honed by scientists and engineers over decades. Sorting is used by billions of people every day without them realising it. It underpins everything from ranking online search results and social posts to how data is processed on computers and phones, so an improvement at this level is of system significance.

As if people's minds are not blown enough, a mere few days later, Dimitris Papailiopoulos, associate professor at University of Wisconsin-Madison, announced on Twitter that he has successfully prompted GPT-4 to discover the same breakthrough Alphadev did. This stirred up a frenzy on the social media platform, eventually catching the attention of its eccentric billionaire owner, Elon Musk. The fact that two different AI were able to discover this new "science" makes it even more exciting as it demonstrates, perhaps the first time since the Enlightenment, a scalable path towards scientific discoveries.

One of the defining features of advanced large language models (LLMs), as is leveraged by Dr. Papailiopoulos in the previous example, is their capacity to mimic a "Chain of Thoughts" or the ability to engage in multi-step inference and decision-making. This powerful functionality mirrors a key attribute that scientists leverage during research: the ability to sequentially reason, make logical connections, and draw nuanced conclusions.

The "Chain of Thoughts" aspect of LLMs essentially allows them to move beyond single, isolated responses and engage in sustained interactions. For example, Auto-GPT can parse complex queries, remember past interactions, reason about context, and generate appropriate responses. This quality enables it to emulate the investigative process inherent to scientific work, thereby opening new frontiers of possibility.

The scientific method, rooted in empiricism and iterative learning, forms the philosophical backbone of scientific investigation. It consists of generating hypotheses, designing and conducting experiments, analyzing data, and then either confirming, refuting, or refining hypotheses based on the findings. Through their "Chain of Thoughts" capabilities, LLMs echo this systematic approach, being able to 'reason' through a problem, consider different facets, analyze, and generate informed responses.

An important caveat must be noted here. LLM can't genuinely hypothesize or question; they merely simulate these processes based on patterns gleaned from their training data. Despite this, LLMs' 'thought simulation' presents exciting possibilities. Imagine an LLM aiding in hypothesis generation or experimental design by drawing upon vast amounts of scientific literature to suggest novel combinations or perspectives. Or consider the role of an LLM as an indefatigable reviewer, combing through reams of data for patterns that might elude human scrutiny.

The implications are significant. The role of the scientist may evolve, as tasks that were once solely in the human domain become shared with advanced AI tools. This raises profound questions about the nature of scientific investigation and the human/AI partnership. How will the scientific method be adapted or expanded in this new paradigm? What checks and balances should be in place to ensure scientific rigor and integrity as we increasingly rely on these tools? And finally, how will the synergy between human and machine-led inquiry catalyze the next wave of scientific discovery?

These are exciting times. The advent of LLMs beckons us into an era of unprecedented scientific collaboration between humans and machines. As we grapple with these profound philosophical questions and implications, we are not merely spectators but active participants in defining the future of scientific exploration.

In recent years, one area that has attracted significant attention and capital is the development of large ML models, especially after ChatGPT’s phenomenal rise to fame. These models require massive computing resources and specialized expertise, leading to high OpEx in the form of compute costs and headcount cost. The OpEx of AI startups can run into hundreds of millions each year, as is shown by DeepMind (~$500m a year per its official financial record) and OpenAI. This unprecedented burn-rate would pose a large strain on a company’s financial health and its survivability.

In order to adapt to this new reality, there's a growing trend of investors treating Operating Expenditures (OpEx) as the new Capital Expenditures (CapEx). This shift is due to the unique financial dynamics of the AI industry, where startups often require massive amounts of computational resources and expertise to develop cutting-edge products and services.

While some AI startups have demonstrated impressive technological feats in natural language processing, computer vision, and reinforcement learning, the path to monetizing these achievements at meaningful scale remains ellusive at best. As such, the influx of capital into this space raises questions about the sustainability of this trend and the potential for another investment bubble.

OpEx without Revenue is no different from CapEx

One of the hallmarks of AI startups is their ability to consume vast amounts of OpEx, often into the hundreds of millions per year. In traditional industries, this level of expenditure would typically be associated with CapEx, where companies invest in long-term assets to drive future growth. However, AI startups can burn through these funds without generating substantial revenue, as they invest heavily in research, development, and talent acquisition to push the boundaries of artificial intelligence.

This cash flow behavior highlights the need to reassess how we categorize these expenses and evaluate AI startups' financial performance, which in turn determines an investor’s real return from betting on these projects.

To better understand the dynamics at play, we can draw comparisons to the ridesharing market, which saw a similar influx of capital in the early 2010s. Investors poured billions into Uber and Lyft, with the expectation that their rapid growth would lead to market dominance and ultimately, profitability. However, despite impressive growth in both user base and revenue, both companies have yet to achieve consistent profitability. For Uber and Lyft, the bulk of capital went towards subsidizing rides in an attempt to win market share and achieve economies of scale.

While early investors profited from subsequent funding rounds and IPOs, the majority of capital invested in these companies yielded little, if not negative, returns. This demonstrates the risks associated with investing in high-growth, high-burn rate ventures, where the path to profitability remains uncertain.

Why Cloud Expenditure is more CapEx than OpEx in the AI Industry

Returning to the AI space, we observe that the high OpEx associated with training large ML models could be viewed as a de facto CapEx. The compute costs are significant and unavoidable, as the quality of ML models is directly tied to the size of model and data, which translates to computational power used in the training & inference process.

The analogy bears even more sense when one observe the predictability of AI OpEx. From an accounting perspective, the point of seperating different expenses into buckets is precisely because of similarity of spending “pattern” within the bucket. CapEx, for example, is expected to be:

• planned ahead and paid upfront

• large amount

• eligable for depreciation (which makes the P&L looks much better)

The traditional OpEx on the otherhand, is paid on-demand and generally at smaller amount. However in reality, many AI startups choose to lease cloud GPUs annually to save cost, instead of paying on-demand (much higher per unit cost), or waiting for spot instance (almost no capacity available since everyone is grabbing as many GPU as they can get their hands on).

These cloud expenditures often have a predictable and upfront cost structure, making them more similar to traditional CapEx than variable OpEx. In addition, the intense competition for top ML talent has driven up salaries and increased the cost of hiring and retaining skilled researchers and engineers. In a tight labor market today, companies seem to prefer “hoarding” talent over headhunting them as needed. This is especially true in the AI market where the supply of top talents are almost inelastic with top schools like Stanford, CMU only churning out numbered PhDs and dropouts each year.

How to invest SMART in a hype?

To be clear, I have no doubt that generative AI and large ML models will reshape how we work and live at a fundamental level, during which trillions dollar worth of value will be created. And I have been seed investing in this space since 2016. Yet, as an investor myself, I have personally seen enough “good ideas” and “good projects” end up in flame due to poor planning and poor execution.

As an AI investment thesis, I’m pretty confident that if one were to put together a portfolio of MSFT/Nvidia/Google weighted by their market cap, this portfolio will outperform at least 75% of AI-themed pe/vc funds in the next 10 years.

I can say this with some confidence cuz a similar portfolio strategy (weighted FAANG) indeed outperformed most TMT-themed pe/vc during 2010-2020; another similar portfolio strategy (weighted Alibaba, Tencent, Baidu) also outperformed most China funds during 2005-2015; same is true for crypto (weighted BTC/ETH vs Crypto funds).

One should also note that this is, by no means, the first AI hype. In the early 2010s, when deep learning first became a thing, leading AI researchers lined up to form paper companies with no visible business plan and was able to raise billions collectively. One of these companies recently got acquired at less than 1/10 the valuation at which they raised their seed round.

The lesson here is elementary (no pun intended).

Summary

AI startups often claim to be "lean" due to their low CapEx requirements in the traditional sense. However, this argument can be misleading, as it overlooks the massive OpEx commitments these companies undertake. In some ways, this perspective resembles the accounting tricks used by companies that capitalize expenses through depreciation to make their Profit & Loss (P&L) statements look more favorable. To gain a more accurate understanding of an AI startup's financial health, it's essential to peel back these accounting “tricks” and focus on the real cash flow dynamics that drive these businesses.

Ultimately, the sustainability of the AI startup ecosystem will depend on the development of viable business models that can monetize the impressive technological advancements in this space. As the market matures and the hype subsides, the true winners in this sector will be those who can successfully navigate the challenges of high OpEx and create lasting value for their investors and customers alike.

There is overwhelming studies that concluded that human activities such as the consumption of fossil fuels have created long lasting environmental challenges including air pollutions. According to the WHO, air pollution kills at least 7 million people a year, while a recent study estimated 8.7 million early deaths a year from coal, oil and gas burning – 20% of all deaths. Among the pollutants from fuels, Nitrogen Oxides (NOx) are particularly notorious for causing the infamous Los Angelas Smog in the 1950s~70s, which subsequently pushed Congress to establish an entirely new regulatory entity now know as the EPA.

Given the severty of the issue, scientists have sought out to tackle the challenge. However, fundemantal understanding of NOx's reactions is still lacking, preventing more systematic solutions from being developed. In particular, the mechanism of NOx's reactive uptake, which dictates it's concentration in the air, was still a mystery, until very recently. The Limmer Research Group at UC Berkeley revealed in their paper published on 《Science》, a method of using AI to unveal the reactive uptake mechanism of NOx.

We have had the previlege to sit down with Dr. Limmer to talk about his work. Here are some highlights.

Q: Why NOx reactive uptake is an important and challenging topic?

There is an outstanding challenge in atmospheric chemistry associated with where all the the NOx compounds go. Despite a ton known about how many molecules the N2O5 get irreversibly taken up by a droplet on a macro-level, the molecular details are still lacking.

This is a challenging system to study. Because the reaction occurs quickly. N2O5 is a gaseous molecule, it doesn't have much time to get into the bulk of aerosol, into the interior. Which means that likely, a lot of the reactivity happens at the interface.

This is what really makes it difficult. To begin with, reactive system requires quantum mechanics simulation, which is costly. On top of that, representing reaction at an interface, adds additional challenges. Because it requires a big enough to differentiate between an interfacial region and a bulk liquid behind it. Just needing a big enough system and needing a quantum mechanical description of that system really made it intractable. They just couldn't be done using traditional methods.

Q: How is the experience of using AI for Science?

There are two distinct problems within this challenge. One is the representation problem (aka. how do you describe the problem setting to a computer), the other is the rare event problem (aka. if the event you want to observe is rare, then it would take a computer a prohibitively long time to simulate).

If you want to study chemistry at an interface, you have to worry about representing a big enough system. I had known about Deep Potential Molecular Dynamics (DeePMD) since I had been at Princeton, right as it was starting to be kind of work done, by Weinan E and his group. This was the right tool, because if you can construct a neural network, you can really get to a very large system, very effectively.

At the same time, you also have to wait for the reactive event to occur. In most of chemistry, reactive event on atomic level happens extremely rare, relative to molecule’s own movement. A reaction might take nanoseconds, whereas a typical time scale for a molecule to diffuse its diameter is a picosecond, 1000x faster. That gets worse and worse the rarer of the event that you want to see.

In this study, it is observed that the wait time for reaction is long because the molecules move slowly and they have to get in just the right confirmation in order for that reaction to occur. But when they get in the right position, it occurs very quickly. We adopted a workaround by putting in a bias for molecules to go where they want to react, and then account for that bias in a statistical sense, in order to have something which is ultimately an unbiased system and of how long it would take for a reaction to occur.

The methodology yielded orders of magnitude improvement in efficiency. The savings for the calculations we did was something like, even including training and everything, like a factor of 1,000. In reality, my postdoc spent 2 years training these models and running in all the calculations. So the traditional method would have taken 1,000 years, right? Something of that order. I maybe if if I was really lucky and got access to the whole supercomputer, maybe you cut that down by a factor of ten, but it's still a hundred years.

Q: Intuitively how is using AI for science different from other popular AI applications (such as AlphaGo)?

So the the strategies one uses in ai for something like AlphaGo, there's many steps before you get an indication of success. In AlphaGo, you need to train a computer to develop some strategy. And then only after many moves, do you know if that strategy is gonna pay off. In some sense, science is a much simpler kind of problem for AI.

Loosely speaking, physics is acting like a guardrail for AI so it does not wander off. In this study, researchers were able to put in a lot of physical constraints to make AI’s training much easier. For example, the forces between atoms shouldn't matter on things like the absolute coordinate system, so one can have a transitional or rotational invariance built into the machine learning scheme. Another important contrain is “locality”. That particles only interact in a finite range means that AI don't have to learn a global surface, which drastically reduced the learning difficulty.

For most problems, the key might be finding the middle ground somewhere between using the flexibility afforded by a neural network and then bringing in physical intuition.

Q: What's next for AI in Environmental Science

There's an immediate next step. Our work indicate that longstanding perspective was wrong — that the majority reaction really does happen at the interface, not in the bulk.

While exciting, it comes with a big challenge. Previous works derived from the old perspective now need to be reestablished and reevaluated. The short-term goal is to figure out how to generalize this perspective to the full complexity of the atmosphere.

The longer-term goal, at least on the DeePMD side is to keep fitting reactive models because they really changed the game for what we can study.

From my perspective, I was mostly motivated by the basic physics and chemistry of how reactions could be different at interfaces. For example, cellular environments are almost all interfaces. There are proteins and membranes and all sorts of things that are catalyzing is in reactivity. We haven't had the tools to study any of those things. So part of my group is starting to deploy these same sorts of methodologies for biochemical reactions. Like, how does the cells start manufacturing proteins? That's something that would be very difficult to stimulate without neural network force fields.

--

Final thoughts

Beyond Dr. Limmer's work, the adoption of AI for Science is helping shed light on various previously prohibitive areas, such as:

§   Particle Physics, Earth Science (high temperature, pressure and other extreme conditions）

§   Protein folding, high entropy alloy (complexity due to scale)

§   Battery solid electrolyte interphase, catalysis (complexity due to interface)

These works might seem far away from our lives, but they are in fact critical for building a prosperous, sustainable green future by enabling more efficient energy generation & storage, faster drug discovery and more enduring and environmentally friendly materials & products. With AI for Science, we might finally get the promised flying cars, instead of just the 144 characters.

AI is Leaving a lot of Knowledge on the Table

“We wanted flying cars, instead we got 140 characters.” Peter Thiel said in 2013.

The statement was generally true even years later. Despite the fast evolution of technology and the huge amount of wealth it has created, it seems that science has been disporportionally under exposed to such development. The best minds on earth have been pouring their talent onto making more excuses to show us ads for the past two decades, why not harness their collective wisdom to advance our understanding of the universe?

What is keeping us from using AI to solve difficult science challenges? And how do we address these impediments? As it turns out, the answer is both intuitive and profound.

Intuitively speaking, imagine if you are an art student and all of the sudden you are required to take an advanced calculus exam. Mostly likely you will not fare well. The same issue happened with first-gen AIs, which have been ranking pages, watching Youtube videos and categorising cat photos for the most part.

Slightly more rigorously speaking, the path of adopting AI for science relies on two fundamental elements:

1. the availability of high-quality domain-specific data (“the data challenge”); and

2. the ability to preserve physics-law during AI training (“the modeling challenge”).

To illustrate the data challenge, let’s look at the development trojectory of computer vision. One could argue that it wasn’t until ImageNet before CV started to really take off. ImageNet provided the high-quality data AI relies on for training, yet high-quality scientific dataset is rarely big enough for training, except for a few specific areas (ex. protein, genomics).

The modeling challege greatly impacts the efficiency of training, which in turn, affects the accuracy of the model output as well as the cost of the whole project. For example, if one were to train a model that predicts certain property based on a given object’s crystal structure (expressed in the form of 3D atom coordinates); then one should expect the model to give the same prediction to inputs that are symmetric. Aka, holding the same object upside-down should NOT have changed it’s basic properties. Yet for the early AI models used in CV and NLP, this was hardly taken into considerations. Without an additional construct for preserving phyiscs-laws, one could say that AI is “leaving a lot of knowledge on the table.”

-

The Dawn of AI for Science

The turning point really came around year 2019~2021.

Mostly famously, AlphaFold2 (2021) demonstrated that with careful construct, AI can achieve truly meaningful result in science areas where there is a lot of data (PDB in this case). AlphaFold2 made it possible to predict protein structure accurately with computation, which paved the way towards massively cheaper and faster drug development in the future.

Meanwhile, a Princeton team led the development of Deep Potential Molecular Dynamics, which uses AI to approximate the behavior of particles. The team made use of quantum physics to generate high-fidelity simulated data, and construct a neural network that explicitly preserves physical laws during training. They are the first to successfully simulate the movement of billions of atoms accurately. In recognising this achievement, their work is awarded the ACM Gordon Bell Prize in 2020.

During the same period, Nvidia also announced their physics-informed ML framework, Modulus, which focus around computational fluid dynamics and have shown promising results in weather forecast and other areas.

-

From Physics-informed to Physics-enforced

For building physical models from data, machine learning also provides new ideas. Merely fitting data is not enough. Physical constraints need to be combined and training should use representative data, so that the model is interpretable and guarantees extrapolability. This brings hope to many problems that are hard to model starting from general physical principles. As long as physical constraints are satisfied, such models can achieve reliability comparable to general physical models.

1. Classical AI: From the perspective of "classical AI", improving a model's physical fidelity mainly comes from enhancing the quality of its training data. This is the weakest form, simply embedding physical knowledge into the training data itself. For example, generating data satisfying conservation laws or known symmetries for training. Such end-to-end utilization of data for modeling is fast and easy, but cannot guarantee the model will obey physical laws during inference.

2. “Physics-informed”: A soft form is indirectly strengthening physical laws through the loss function. For example, Physics-informed Neural Network (PINN) incorporates differential equations into part of the loss function, guiding parameter optimization. PINNs embed differential equations and other physical constraints into the neural network structure to approximate functions containing physical laws. The inputs to a PINN include independent variables x and parameters λ, and the output is the solution u(x,λ). Network parameters are trained by minimizing a loss function consisting of two parts: 1) Data fitting - using network output u(x,λ) to approximate given data; 2) Physical constraints - taking derivatives of u(x,λ) and substituting into differential equations to require satisfying the equations. This is implemented via automatic differentiation. After training, the lowered loss function means an u(x,λ) satisfying both data and physical constraints is found. Variational methods also belong here, optimizing a functional (such as action in classical mechanics or free energy in statistical mechanics). Such forms can partially constrain the model, but will not rigidly require obeying physical principles.

3. “Physics-enforced”: A stronger form directly builds fundamental physical laws into the model architecture. Using descriptors preserving problem symmetries is a typical example. Another is constructing Hamiltonian or Lagrangian neural networks with structures respecting certain physical conservation laws like energy. This form maximally ensures model behaviors conform to physical principles, but also restricts model expression. In DeePMD, researchers cleverly incorporated some key physical constraints: 1) Translation invariance - the total energy of a physical system does not depend on absolute atomic positions. This is achieved by expressing the system energy as a sum over atomic energies, each determined by the atom's local environment defined relative to the atom, thus ensuring translational invariance. 2) Rotation invariance - the total energy should not change under system rotations. DeePMD implements this by using rotationally invariant local descriptors. 3) Permutation invariance - swapping atom indices should not change the total energy. DeePMD also guarantees this through summing atomic energies. These invariances are key components of the DeePMD design. The model is trained on potential energy surfaces from quantum mechanics calculations. The invariances help ensure the learned model can generalize well to new systems. Without them, the model would struggle to predict properties of systems different from the training data.

It should be emphasized that stronger physical constraints are NOT “better” by default. It depends on the specific factors for each case (such as data quality, and the end user’s actual expectation). For example, AlphaFold can be considered an algorithm between L1 ~ L2, yet thanks to the extremely high quality of the PDB data source, its training is very successful, with excellent inference performance and efficiency. Meanwhile, introducing strong L3 constraints sometimes decreases model trainability, impacting time and cost. When constructing AI for Science algorithms for specific scenarios, pragmatism is always a good idea.

-

Final Remarks: New Paradigm & Early Adopters

“In the history of science, there were two periods of time that made the most impact for applied mathematics. The first was the time of Newton, during which it was established that mathematics should be the language of science. The second was the time of von Neumann, during which it was proposed that numerical algorithms should be the main bridge between mathematics and science.

Now the third time is at the horizon, a time when all the major components of applied math are in place, to form the foundation of not only interdisciplinary scientific research but also exciting technological innovation.” — Weinan E

We are seeing AI4S gaining adoption by players from a wide range of disciplines and industries. Very recently, Microsoft also announced a huge bet on AI4S by creating a dedicated AI4S team within MSR. Eric Schmidt also pledged over $100m for AI for Science scholarship. Conservatively speaking, it might still be a while before the AI4S community finished building the infrastructure layer (ex. the “ImageNet”, “TensorFlow” and “Matlab” of AI4S). But we are already looking at promising progress in pharmaceuticals, energy materials and more sectors.

-

Science and Technology are often used interchangably in everyday situations. One could argue that it is because they are the opposite sides of the same coin. Over-simplified, Science is the ‘reduction’ of nature to rules, and Technology is the ‘construction’ of nature from rules.

From Newton to the industrial revolutions to the emergance of computers and Internet, paradigm-shifting technologies had always emerged quickly after the establishment of the corresponding science. These include everything we take for granted as part of modernity: internal combustion engines, light bulbs, and Wi-Fi, to name a few.

In the first half of 1900s, quantum physics was established and was widely considered to be the ultimate science we need to explain everything — from the basic particles all the way to lifes and galaxies. However, things didn’t go as planned. 100 years after Schrodinger established the Nobel-winning equation that governs the movement of particles, we still don’t have a perfect solution to convert this science into technologies — we still cannot rationally design a ligand for a specific diasease (despite trillions of dollar in healthcare research); we still can’t design a viable lithium metal battery (even after Sony commercialised Lithium-ion battery 30 years ago); and we still don’t have fusion energy (despite being ’50 years away’ for more than 50 years).

Billions of dollars are spent on simulation softwares every year, which is supposed to help ‘construct’ nature from science; yet researchers still needs hundreds of billions dollar more to spent on wet labs to generate experimental results and ‘reduce’ them into patterns/rules. In areas such as drug design and material science, research is still largely done by “trial and error” not unlike the days of Thomas Edison a century ago.

Why the Gap?

In PW Anderson’s article ‘More is different’ in 1972, he argues that finding the laws that govern nature is not sufficient for understanding nature. ‘Reductionism’ leads us to these laws, but that does not make the opposite path (‘constructionism’) trivial, or even feasible. Another Nobel Laureate, Roald Hoffmann in his recent essay “Simulation vs. Understanding”, echoed the same notion.

As Paul Dirac famously said

“The underlying physical laws necessary for the mathematical theory of a large part of physics and the whole of chemistry are thus completely known, and the difficulty is only that the exact application of these laws leads to equations much too complicated to be soluble.”

Even with powerful simplications like Kohn-Sham, solving Schrodinger equation for a reality-relevant system (typically with >100k atoms) have proven to be near-impossible even with the largest supercomputers in the world.

On the macro level, the pain is equally accute. Imagine if it takes 2h to generate a forecast for the weather next hour. Then this forecast, however accurate, would be de facto useless. This, unfortunately, is still largely the reality today, despite billions of dollar spent by governments on large computation infrastructures.

“More is different” means stacking up CPUs/GPUs will hit the wall of diminishing returns before yielding much meaning results.

As scientists looking for more powerful mathematics tools, AI emerged as a promising candidate. One could observe that a common challenge across science disciplines is efficiently solving high dimensional equations (Schrodinger’s equations, Navier-stokes equations, for example), and AI (machine learning) has, in recent years, demonstrated such capability in similar challenges like Computer Vision (lots of pixels -> lots of dimensions). The analogy is clear.

So, it is only intuitive to ask "can we use AI to fit high dimensional scientific equations", aka, "can we teach science to artificial intelligence".

Stay tune for the next post on what is AI for Science