It is non-stop in the news, every week it pops up in another corner, AIs based on Deep Neural Networks, so i will give it a try to write a lill, biased article about this topic...

The brain

The human brain consists of about 100 billion neurons, as much as stars in our galaxy, the Milky Way, and each neuron is connected via synapses with about 1000 other neurons, resulting in 100 trillion connections.

For comparison, the game playing AI, AlphaZero, by Google Deepmind used about 50 million connections to play chess on super human level.

The inner neurons of our brain are connected via our senses, eyes, ears, etc, with the outer world.

One neuron has multiple, weighted inputs and one ouput, if a certain threshold of input is reached, its output is activated, the neuron fires an signal to another neuron.

The activation of the synapse is an electrical and chemical process, neurotransmitters can restrain or foster the activation potential, just consider the effect alcohol or coffee has to your cognitive performance.

Common artificial neural networks do not emulate the chemical part.

The brain wires these connections between neurons during learning, so they can act as memory, or can be used for computation.

The "von Neumann" architecture

Most nowadays computers are based on the von Neumann architecture, they have no neurons or synapses but transistors.

The main components are the ALU, Arithmetic Logic Unit, memory for program and data, and various inputs and outputs.

Artificial Neural Networks have to be built in software, running on these von Neumann computers.

Von Neumann said that his proposed architecture was inspired by the idea of how the brain works, memory and computation. And in his book, "The Computer and the Brain", he gives an comparision of computers and the knowledge about biological neural networks of that time.

Dartmouth

First work on ANNs were published already in the 1940s, and in 1956 the "Dartmouth Summer Research Project on Artificial Intelligence" was held, coining the term Artificial Intelligence, and marking one milestone in AI. The work on ANNs continued, and first neuromorphic chips were developed.

AI-Winter

In the 1970s the AI-Winter occurred, problems in computational theory and the lack of compute power needed by large ANNs, resulted in cutting funds, and splitting the work into strong and weak AI.

Deep Neural Networks

With the rise of compute power (driven by GPGPU), further research, and Big Data, it was possible to train faster better and larger networks in the 21st century.

The term Deep Neural Networks, for deep hierarchical structures or deep learning techniques was coined.

One of the first and common usage for ANNs was and is pattern recognition, for example character recognition.

You can train a neural network with a set of the same, but different looking character, with the aim that the ANN will recognize the same character in various appearances.

With a deeper topology of the neural network, it is possible to identify for example pictures of cars with different net layers for color, shape etc.

The Brain vs. The Machine

A computer can perform fast arithmetic and logical operations, therefore the transistors are used.

Contrary, the neural network of our brain works massiv parallel.

The synapses of the human brain are clocked with 10 to 100 hertz, means they can fire to other neurons up to 100 times per second.

Nowadays computer chips are clocked with 4 giga hertz, means they can compute 4 000 000 000 operations per second per ALU.

The brain has 100 billion neurons, 100 trillion connections and consumes ~20 watt, nowadays biggest chips have 12 billion transistors with an usage of 250 watt.

We can not compare the compute power of an brain directly with an von Neumann computer, but we can estimate what kind of computer we would need to map the neural network of an human brain.

Assuming 100 trillion connections, we would need about 400 terabytes of memory to store the weights of the neurons. Assuming 100 hertz as clock rate, we would need at least 40 petaFLOPS (floating point operations per second) to compute the activation potentials.

For comparison, the current number one high performance computer in the world is able to perform ~93 petaFLOPS, has ~1 petabyte memory, but an power consumption of more than 15 megawatt.

So, considering simply the energy efficiency of the human brain,
i give -1 points for the Singularity to take off.

Comments

SrdjaTuesday, 25 February 2020Followup - since some time GPUs are used to train and infere ANNs. These have more compute horse power in this concern than general purpose CPUs. Recently there was another shift to TPUs, Tensor Processing Units, these perform matrix-math, optimized to speed up ANNs. The Nvidia DGX2 single node server contains 16 GPUs with TensorCores and achieves up to 2 petaFLOPS. If we consider that Moore's Law has still something in pipe, we could extrapolate that within 10 years the barrier for an AGI/ASI, an Super-Intelligence, is reached...

SrdjaMonday, 02 March 2020Second followup - just to clarify, our current models of artificial neural networks we use in applied computer science do currently not cover the whole complex of human brains, we just run layers of neurons connected to other layer of neurons and do not simulate neurotransmitters, ion channels, magnetite, etc.

SrdjaMonday, 12 October 2020Third followup - in this article's calculation with an estimated 40 petaFLOPS for an ASI (Artificial Super Intelligence) or AGI (Artificial General Intelligence) I took not into account that we can create an intelligence based on written text only, no video or audio or other sensors in need which for sure consume a broad portion of a human's brain capacity...

SrdjaSunday, 17 November 2024Fourth followup - the estimation should be 400 terabytes and 20 petaFLOPS for 100 trillion connections with 100 Hz neuron firing rate. The memory required depends on precision of the data type, from FP4 over FP32 to FP64, so lower bound is currently 50 terabytes and upper bound is 800 terabytes. The needed computing power measured in FLOPS depends on the architecture of the neural network, we will need at least 2 FLOP (multiply+accumulate) per connection resp. weight resp. parameter, so 20 petaFLOPS represents the lower bound.

SrdjaSunday, 17 November 2024Current data center type AI GPUs offer over 100 gigabytes RAM and deliver over 1 petaFLOPS matrix compute, with up to 8 GPUs in one server node and 256 GPUs in a pod of nodes.

SrdjaMonday, 18 November 2024If we assume 8 GPUs per node and 16 nodes via MoE parallelization used for NN inference, further the pace of fab process, further the LLM scaling progress, one could estimate that by 2025 we have the hardware present for an AGI and by 2026 a running AGI, futher by 2027 we have the hardware for an ASI and by 2028 an running ASI.

SrdjaMonday, 18 November 2024Ofc, one could say that is just a matter of how much money you are throwing at it, and it is questionable if we have enough human generated data for training and/or can switch to reinforcement learning.

Pages

Standing

Recommended

Posts by year

Tag Cloud

On Artificial Neural Networks