- Python integers, as opposed to numpy integer types, are represented with a flexible number of bits: `sys.int_info` ➔ `bits_per_digit=30, sizeof_digit=4, default_max_str_digits=4300, str_digits_check_threshold=640`
- they are called "long" or "sperlong" integers, because they can have arbitrary size. Low level implementation explained:
- [all types](https://float.exposed/) with [explanation](https://ciechanow.ski/exposing-floating-point/)
- Docs with more details: [What Every Programmer Should Know About Floating-Point Arithmetic or Why don’t my numbers add up?](https://floating-point-gui.de)
- Docs with the gory details, a.k.a. the floating point bible: [What every computer scientist should know about floating-point arithmetic](https://doi.org/10.1145/103162.103163)
- **strings**:
- UTF8 encoded, flexible width from 1B (byte) to 4B (bytes): 1,112,064 Unicode characters (code points)
- ASCII: 7 bits (fits in one byte), 127 characters ➔ [ASCII table](https://upload.wikimedia.org/wikipedia/commons/2/26/ASCII_Table_%28suitable_for_printing%29.svg)
- Primer on CPU (x86_64) architecture and the memory hierarchy:
- CPU registers ≈ 160 (plus another ~500 model specific), latency: 0 cycles, capacity: 8 bytes
- x86-64 instruction set, with ≈ 2000 instructions with mnemonic (plus an unknown number of "undocumented" instructions ~ 10k)
- (single instruction, mutiple data [8 or 16 data units]) SIMD CPUs
- L-Caches: L1/L2/L3, with cache lines of 128B, latencies: 1-40 cycles, capacity: ~KB and ~MB
- Main memory: RAM pages 4KB or 64KB, latency: 50~100 cycles, capacity ~GBs
- Storage (local disks): disk transfer blocks 4KB to 64MB, latency: 0.1ms (300k cycles), capacity: ~TBs
- Remote Storage (network): typically limited by ethernet connection (1-10 GB/s), latency: 10~100 ms, capacity: ∞
- Understand the trade-offs involved:
- **capacity** measured in T/G/M/K/B
- **latency** ≈ (time-when-data-available-on-output-pins – time-when-data-requested) ➔ measured in nanoseconds or in (CPU) cycles
- **bandwidth** ≈ clock frequency × data-transfer/tick × bus-width (in bytes) ➔ measured in T/G/M/K/B per second (this is what is usually advertised as the **speed**)
- The gory details about the memory hierarchy: [What Every Programmer Should Know About Memory](https://www.akkadia.org/drepper/cpumemory.pdf) by the notorious Ulrich Drepper
# …and what about the GPU?
![GPU vs CPU architecture](GPUvsCPU-architecture.png)
- A GPU has many (in the order of hundreds) SIMT (single instruction, multiple thread) cores, so called SMs (Streaming Multiprocessors), each one with local L1 and shared L2 caches, and shared RAM (due to to the high parallelism, with huge bandwidth, in the order of ~1TB/s
- The SMs are specialized on data types. In order of abundance, the following data types are supported: int8, int32, int64, float16, float32, float64
- Performance depends on:
- memory bandwidth: usually higher than CPU's RAM
- "math" bandwidth: usually higher than CPU's, but much more limited in capability; for example branches (if/else) are expensive
- latency: usually much higher than CPU's ➔ the more parallel threads are run the less the price of high latency is paid (latency "hiding")
- spatial locality is extremely critical
- A portion of the GPU-RAM is accessible to the CPU ➔ the GPU performs the copies
- The PCI-Bus bottleneck: data needs to flow from main (CPU) memory to GPU memory and back!
- Problems on a cluster: the GPU does not really support simultanous multiple users payloads!
# Computer Architecture (a concrete example)
My Laptop:
![Graphical representation](topology.png)
- Lenovo - T14 Gen 4
- CPU i7-1365U:
- 2× "performance cores" (Intel Core) max 5.20 GHz (0.19 ns/cycle) with Hyper-Threading