“As our circle of knowledge expands, so does the circumference of darkness surrounding it.”
- Albert Einstein
There are two types of goals: instrumental and terminal:
Terminal: end game, no justification needed
Instrumental: helps you achieve your terminal goals
The Orthogonality Thesis is a relationship between intelligence and motivation, and states that any level of intelligence can be compatible with any level of motivation.
Bostrom states that there are 3 main ways one could go about predicting goals of a superintelligence:
design, inheritance, and instrumental convergence
Instrumental Convergence is an important concept and relates how you could predict what paths a superintelligence might take to achieve its terminal goals.
There are three main ways for a superintelligence to “fail”
Perverse Instantiation, Infrastructure Profusion, and Mind Crimes
These are called malignant failure modes and are a huge problem when wanting to articulate or program a goal.
View Full Lecture PDF Below: