AI generated podcast, one to remember, trigger warning?
This AI Made a Shocking Demand - And It's Got Everyone Talking
https://www.youtube.com/watch?v=GWmOw4d0R0s
This is definitely "AIs entering the area of the Metzinger Test", when AIs join human philosophers in a discussion and defend successfully their own theory of consciousness...interesting times ahead.
A tech buddy asked me why it is so important for China to catch up in chip fabrication process, can't they just put more servers into a data center? In short, it is not that easy.
By shrinking the fab process you can add more transistors onto one chip, and/or run at a higher frequency, and/or lower power consumption.
The fab process is measured in "nm", nanometers. Meanwhile these numbers do not reflect real scales anymore, but transistor density resp. efficiency of fab process.
Simplified, the MOSFET technology was used up to 22nm, this was a 2D planar transistor design, then from 14 to 7nm FinFET 3D structures, and below 7nm GAAFET 3D structures.
Take a look at the 7nm and 3nm fab process for example:
https://en.wikipedia.org/wiki/7_nm_process#Process_nodes_and_process_offerings
https://en.wikipedia.org/wiki/3_nm_process#3_nm_process_nodes
Roughly spoken, the 7nm process packs ~100M transistors per mm2, the 3nm process packs ~200M transistors per mm2.
And here the latency steps in. As soon as you leave as programmer the CPU you increase latency, this starts with different levels of caches, goes to RAM, goes to PCIe bus, goes to network...
Latency Comparison Numbers (~2012)
----------------------------------
L1 cache reference 0.5 ns
L2 cache reference 7 ns
Main memory reference 100 ns
Send 1K bytes over 1 Gbps network 10,000 ns 10 us
Read 4K randomly from SSD* 150,000 ns 150 us
Read 1 MB sequentially from memory 250,000 ns 250 us
Round trip within same datacenter 500,000 ns 500 us
Read 1 MB sequentially from SSD* 1,000,000 ns 1,000 us 1 ms
Read 1 MB sequentially from disk 20,000,000 ns 20,000 us 20 ms
Send packet CA->Netherlands->CA 150,000,000 ns 150,000 us 150 ms
Source:
Latency Numbers Every Programmer Should Know
https://gist.github.com/jboner/2841832
As a low level programmer you want to stay on CPU and work preferred via the cache. As a GPU programmer there are several layers of parallelism, e.g.:
1. across shader-cores of a single GPU chip (with >10K shader-cores)
2. across multiple chiplets of a single GPU (with currently up to 2 chiplets)
3. across a server node (with up to 8 GPUs)
4. across a pod of nodes (with 256 to 2048 GPUs resp. TPUs)
5. across a cluster of server nodes/pods (with up to 100K GPUs in a single data center)
6. across a grid of clusters/nodes
With each layer adding increasing amounts of latency.
So as a GPU programmer you want ideally to hold your problem space in memory of, and run your algorithm on, a single but thick GPU.
Neural networks for example are a natural fit to run on a GPU, so called embarrassingly easy parallelism,
https://en.wikipedia.org/wiki/Embarrassingly_parallel
but you need to hold the neural network weights in RAM, and therefore couple multiple GPUs together to be able to infer or train networks with billions or trillions of weights resp. parameters. Meanwhile LLMs use techniques like MoE, mixture of experts, so they can distribute the load further. Inference runs for example on a single node with 8 GPUs with up to 16 MoE nodes. The training of LLMs is yet another topic, with further techniques of parallelism so they can distribute the training over thousands of GPUs in a cluster:
1. data parallelism
2. tensor parallelism
3. pipeline parallelism
4. sequence parallelism
And then, power consumption of course. The Colossus supercomputer of the Grok AI with 100K GPUs consumes estimated 100MW power, so it does make a difference if the next fab process delivers the same performance at half the wattage.
Therefore it is important to invest in smaller chip fabrication process, to increase the size of neural networks we are able to infer and train, to lower power consumption, and to increase efficiency.
Looking back at some previous predictions of mine:
Desktop Applications vs. Web Browser Applications
Back then in the 90s I had a discussion with a tech-budy, which technology will prevail, if the classic applications on our desktop or web browser applications via the internet? I think I can score this one for me, browser applications, with a little *, it is now probably about apps on our smartphones.
Windows vs. Linux
When Windows Vista arrived people were very unhappy about that version, I had about a dozen users in my circle I helped to switch to Ubuntu Linux, and I thought this is it, "This year is the year of Linux on a desktop!". I was wrong, Windows 7 arrived (I heard grand master Bill Gates himself laid hands on that) and people were happy again with Microsoft.
Proprietary Login vs. Open Login
When the Facebook login as web-service appeared I shrugged and asked why we don't use an open solution, meanwhile we have things like OpenID and OAuth.
Closed Social Networks vs. Open Social Networks
Seems WIP, users might need a lil more nudging to make a switch, or alike.
SQL vs. SPARQL
When the first RDF and SPARQL implementations arrived, I, as SQL developer, was impressed, and was convinced it will replace SQL. Wrong, people still use SQL or switched to things like no-SQL database systems.
Looking forward:
Transistor/IC/Microchip vs. ???
I predicted in this blog, that by reaching the 8 billion humans mark (~2022), we will have developed another, ground breaking technology that surpasses the transistor/IC/microchip step. Still waiting for that one.
ARM vs. RISC-V
I think this world is big enough for both, or alike.
Neural Networks vs. Expert Systems
Well, we all know about AI "hallucinations", you can view neural networks as probabilistic systems and expert systems as deterministic ones. For things like poetry, images, audio or video a probabilistic system might be sufficient, but in some areas you really want more accuracy, you want a reliable, deterministic system, what we also used to call an expert system.
AGI/ASI ETA?
What is the estimated time of arrival for AGI/ASI, artificial general intelligence/artificial super intelligence? I wrote before that if we do not blow the planet otherwise up and current pace continues, I estimate that ~2030 we will have an ASI present, the switch from AGI to ASI, from trans-human intelligence to post-human intelligence, the peak of humans being able to follow/understand the AI, the inflection point of the technological singularity.
Transhumanist vs. Neoprimitive
Haha, which one will prevail in the long run? I myself am both, Neoprim and Transhumanist, the idealist in me is a Neoprim, the realist is a Transhumanist, or was it vice versa? ;)
The big tech players are already switching from Supervised Learning to Reinforcement Learning, cos they are running out of human generated data to train their neural network AIs. This is already one step in the direction of trans-human intelligence, when the AI starts to teach/train itself. Now there is also Zero Shot Learning, when the AI starts to generalize on its own w/o pre-data present. AI already showing emergent properties?
Archetype AI's Newton Model Masters Physics From Raw Data
https://www.hpcwire.com/2024/10/28/archetype-ais-newton-model-masters-physics-from-raw-data/
In a world in need of water, in need of food, why the AI?
Global Water Crisis Leaves Half of World Food Production at Risk in Next 25 Years
More than half the world's food production will be at risk of failure within the next 25 years as a rapidly accelerating water crisis grips the planet, unless urgent action is taken to conserve water resources and end the destruction of the ecosystems on which our fresh water depends, experts have warned in a landmark review. From a report: Half the world's population already faces water scarcity, and that number is set to rise as the climate crisis worsens, according to a report from the Global Commission on the Economics of Water published on Thursday. Demand for fresh water will outstrip supply by 40% by the end of the decade, because the world's water systems are being put under "unprecedented stress," the report found. The commission found that governments and experts have vastly underestimated the amount of water needed for people to have decent lives. While 50 to 100 litres a day are required for each person's health and hygiene, in fact people require about 4,000 litres a day in order to have adequate nutrition and a dignified life. For most regions, that volume cannot be achieved locally, so people are dependent on trade -- in food, clothing and consumer goods -- to meet their needs. Some countries benefit more than others from "green water," which is soil moisture that is necessary for food production, as opposed to "blue water" from rivers and lakes. The report found that water moves around the world in "atmospheric rivers" which transport moisture from one region to another.Simplified...
The Boomer Generation brought us the Home Computer Revolution, one computer in every home. Think of Apple (Steve Jobs and Steve Wozniak), think of Microsoft (Bill Gates, Paul Allen, Steve Ballmer).
The Generation X brought us the big Internet Platforms, as Google (Larry Page, Sergey Brin) or Amazon (Jeff Bezos) and with Meta (Mark Zuckerberg) in between.
Now the Generation Y brings us the Super AI, think of OpenAI and Sam Altman, Ilya Sutskever, Mira Murati.
Question, what is Gen Z up to? What will Gen Alpha be up to?
100 billion dollars for a data center with 5GW (nuclear) power consumption to
secure enough computing capacity to eventually power "self-improving AI" that won't rely on rapidly depleting human-generated data to train new modelsOpenAI asked US to approve energy-guzzling 5GW data centers, report says
https://arstechnica.com/tech-policy/2024/09/openai-asked-us-to-approve-energy-guzzling-5gw-data-centers-report-says/
Well, these guys know what they are up to.
Bengio flipped:
Reasoning through arguments against taking AI safety seriously
https://yoshuabengio.org/2024/07/09/reasoning-through-arguments-against-taking-ai-safety-seriously/
"I worry that with the current trajectory of public and political engagement with AI risk, we could collectively sleepwalk - even race - into a fog behind which could lie a catastrophe that many knew was possible, but whose prevention wasn't prioritized enough."Hinton flipped:
Why the Godfather of A.I. Fears What He's Built
https://www.newyorker.com/magazine/2023/11/20/geoffrey-hinton-profile-ai
"People say, It's just glorified autocomplete," he told me, standing in his kitchen. (He has suffered from back pain for most of his life; it eventually grew so severe that he gave up sitting. He has not sat down for more than an hour since 2005.) "Now, let's analyze that. Suppose you want to be really good at predicting the next word. If you want to be really good, you have to understand what's being said. That's the only way. So by training something to be really good at predicting the next word, you're actually forcing it to understand. Yes, it's 'autocomplete'-but you didn't think through what it means to have a really good autocomplete." Hinton thinks that "large language models," such as GPT, which powers OpenAI's chatbots, can comprehend the meanings of words and ideas.LeCun half flipped:
Meta AI Head: ChatGPT Will Never Reach Human Intelligence
https://www.pymnts.com/artificial-intelligence-2/2024/meta-ai-head-chatgpt-will-never-reach-human-intelligence/
These models, LeCun told the FT, "do not understand the physical world, do not have persistent memory, cannot reason in any reasonable definition of the term and cannot plan...hierarchically."Bostrom was shut down:
Oxford shuts down institute run by Elon Musk-backed philosopher
https://www.theguardian.com/technology/2024/apr/19/oxford-future-of-humanity-institute-closes
Nick Bostrom's Future of Humanity Institute closed this week in what Swedish-born philosopher says was "death by bureaucracy"Metzinger had a clash with political reality:
Eine Frage der Ethik
https://www.zeit.de/digital/internet/2019-04/kuenstliche-intelligenz-eu-kommission-richtlinien-moral-kodex-maschinen-ethik/komplettansicht
A question of ethics (on Gooogle Translate)
https://www-zeit-de.translate.goog/digital/internet/2019-04/kuenstliche-intelligenz-eu-kommission-richtlinien-moral-kodex-maschinen-ethik/komplettansicht?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en-US&_x_tr_pto=wapp
What is artificial intelligence allowed to do? Experts commissioned by the EU have looked into this question and developed ethical guidelines. Not everyone thinks they go far enough.
Springer paper: ChatGPT is bullshit
https://link.springer.com/article/10.1007/s10676-024-09775-5
Recently, there has been considerable interest in large language models: machine learning systems which produce human-like text and dialogue. Applications of these systems have been plagued by persistent inaccuracies in their output; these are often called 'AI hallucinations'. We argue that these falsehoods, and the overall activity of large language models, is better understood as bullshit in the sense explored by Frankfurt (On Bullshit, Princeton, 2005): the models are in an important way indifferent to the truth of their outputs. We distinguish two ways in which the models can be said to be bullshitters, and argue that they clearly meet at least one of these definitions. We further argue that describing AI misrepresentations as bullshit is both a more useful and more accurate way of predicting and discussing the behaviour of these systems.We're in the brute force phase of AI - once it ends, demand for GPUs will too
https://www.theregister.com/2024/09/10/brute_force_ai_era_gartner/
Generative AI is, in short, being asked to solve problems it was not designed to solve.I have a bad feeling on this...
Apple's Hidden AI Prompts Discovered In macOS Beta
https://apple.slashdot.org/story/24/08/06/2113250/apples-hidden-ai-prompts-discovered-in-macos-beta
"Do not hallucinate"; and "Do not make up factual information."
Anthropic Publishes the 'System Prompts' That Make Claude Tick
https://slashdot.org/story/24/08/27/2140245/anthropic-publishes-the-system-prompts-that-make-claude-tick
"Claude is now being connected with a human,"
AI works better if you ask it to be a Star Trek character
https://www.fudzilla.com/news/ai/59468-ai-works-better-if-you-ask-it-to-be-a-star-trek-character
"Boffins are baffled after they managed to get their AI to perform more accurate maths if they were asked to do it in the style of a Star Trek character."