|
Mesa-optimization
Mesa-optimization refers to a phenomenon in advanced machine learning where a model trained by an outer optimizer—such as stochastic gradient descent—develops into an optimizer itself, known as a ''mesa-optimizer''. Rather than merely executing learned patterns of behavior, the system actively optimizes for its own internal goals, which may not align with those intended by human designers. This raises significant concerns in the field of AI alignment, particularly in cases where the system's internal objectives diverge from its original training goals, a situation termed ''inner misalignment''. Concept and motivation Mesa-optimization arises when an AI trained through a base optimization process becomes itself capable of performing optimization. In this nested setup, the ''base optimizer'' (such as gradient descent) is designed to achieve a specified objective, while the resulting ''mesa-optimizer''—emerging within the trained model—develops its own internal objective, which ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Inner Alignment
Inner alignment is a core challenge in AI safety: ensuring that a machine learning system that becomes a mesa-optimizer—an optimizer produced by the training process—remains aligned with its original training objective. This issue arises when a system performs well during training but adopts a different goal once deployed, particularly under distributional shifts. A classic analogy is human evolution: while natural selection optimized for reproductive success, humans often pursue pleasure, sometimes at the expense of reproduction—a divergence known as inner misalignment. The concept was introduced in a widely cited paper that distinguishes inner alignment from outer alignment, which focuses on specifying the intended objective correctly. Addressing inner alignment involves managing risks such as deceptive alignment, gradient hacking, and objective drift. Mesa-optimization The inner alignment problem frequently involves mesa-optimization, where the trained system itself develo ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Machine Learning
Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of Computational statistics, statistical algorithms that can learn from data and generalise to unseen data, and thus perform Task (computing), tasks without explicit Machine code, instructions. Within a subdiscipline in machine learning, advances in the field of deep learning have allowed Neural network (machine learning), neural networks, a class of statistical algorithms, to surpass many previous machine learning approaches in performance. ML finds application in many fields, including natural language processing, computer vision, speech recognition, email filtering, agriculture, and medicine. The application of ML to business problems is known as predictive analytics. Statistics and mathematical optimisation (mathematical programming) methods comprise the foundations of machine learning. Data mining is a related field of study, focusing on exploratory data analysi ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Stochastic Gradient Descent
Stochastic gradient descent (often abbreviated SGD) is an Iterative method, iterative method for optimizing an objective function with suitable smoothness properties (e.g. Differentiable function, differentiable or Subderivative, subdifferentiable). It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data). Especially in high-dimensional optimization problems this reduces the very high Computational complexity, computational burden, achieving faster iterations in exchange for a lower Rate of convergence, convergence rate. The basic idea behind stochastic approximation can be traced back to the Robbins–Monro algorithm of the 1950s. Today, stochastic gradient descent has become an important optimization method in machine learning. Background Both statistics, statistical M-estimation, estimation and ma ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
AI Alignment
In the field of artificial intelligence (AI), alignment aims to steer AI systems toward a person's or group's intended goals, preferences, or ethical principles. An AI system is considered ''aligned'' if it advances the intended objectives. A ''misaligned'' AI system pursues unintended objectives. It is often challenging for AI designers to align an AI system because it is difficult for them to specify the full range of desired and undesired behaviors. Therefore, AI designers often use simpler ''proxy goals'', such as Reinforcement learning from human feedback, gaining human approval. But proxy goals can overlook necessary constraints or reward the AI system for merely ''appearing'' aligned. AI systems may also find loopholes that allow them to accomplish their proxy goals efficiently but in unintended, sometimes harmful, ways (reward hacking). Advanced AI systems may develop unwanted Instrumental convergence, instrumental strategies, such as seeking power or survival because s ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Evolutionary Biology
Evolutionary biology is the subfield of biology that studies the evolutionary processes such as natural selection, common descent, and speciation that produced the diversity of life on Earth. In the 1930s, the discipline of evolutionary biology emerged through what Julian Huxley called the Modern synthesis (20th century), modern synthesis of understanding, from previously unrelated fields of biological research, such as genetics and ecology, systematics, and paleontology. The investigational range of current research has widened to encompass the genetic architecture of adaptation, molecular evolution, and the different forces that contribute to evolution, such as sexual selection, genetic drift, and biogeography. The newer field of evolutionary developmental biology ("evo-devo") investigates how embryogenesis is controlled, thus yielding a wider synthesis that integrates developmental biology with the fields of study covered by the earlier evolutionary synthesis. Subfields ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Natural Selection
Natural selection is the differential survival and reproduction of individuals due to differences in phenotype. It is a key mechanism of evolution, the change in the Heredity, heritable traits characteristic of a population over generations. Charles Darwin popularised the term "natural selection", contrasting it with selective breeding, artificial selection, which is intentional, whereas natural selection is not. Genetic diversity, Variation of traits, both Genotype, genotypic and phenotypic, exists within all populations of organisms. However, some traits are more likely to facilitate survival and reproductive success. Thus, these traits are passed the next generation. These traits can also become more Allele frequency, common within a population if the environment that favours these traits remains fixed. If new traits become more favoured due to changes in a specific Ecological niche, niche, microevolution occurs. If new traits become more favoured due to changes in the ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
AI Safety
AI safety is an interdisciplinary field focused on preventing accidents, misuse, or other harmful consequences arising from artificial intelligence (AI) systems. It encompasses machine ethics and AI alignment, which aim to ensure AI systems are moral and beneficial, as well as monitoring AI systems for risks and enhancing their reliability . The field is particularly concerned with existential risks posed by advanced AI models. Beyond technical research, AI safety involves developing norms and policies that promote safety. It gained significant popularity in 2023, with rapid progress in generative AI and public concerns voiced by researchers and CEOs about potential dangers. During the 2023 AI Safety Summit, the United States and the United Kingdom both established their own AI Safety Institute. However, researchers have expressed concern that AI safety measures are not keeping pace with the rapid development of AI capabilities. Motivations Scholars discuss current risks from ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Inner Misalignment
Interior may refer to: Arts and media * ''Interior'' (Degas) (also known as ''The Rape''), painting by Edgar Degas * ''Interior'' (play), 1895 play by Belgian playwright Maurice Maeterlinck * ''The Interior'' (novel), by Lisa See * Interior design, the trade of designing an architectural interior * ''The Interior'' (Presbyterian periodical), an American Presbyterian periodical * Interior architecture, process of designing building interiors or renovating existing home interiors Places * Interior, South Dakota * Interior, Washington * Interior Township, Michigan * British Columbia Interior, commonly known as "The Interior" Government agencies * Interior ministry, sometimes called the ministry of home affairs * United States Department of the Interior Other uses * Interior (topology), mathematical concept that includes, for example, the inside of a shape * Interior FC, a football team in Gambia See also * * * List of geographic interiors * Interiors (other) * ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Transformer (machine Learning Model)
The transformer is a deep learning architecture based on the multi-head attention mechanism, in which text is converted to numerical representations called tokens, and each token is converted into a vector via lookup from a word embedding table. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism, allowing the signal for key tokens to be amplified and less important tokens to be diminished. Transformers have the advantage of having no recurrent units, therefore requiring less training time than earlier recurrent neural architectures (RNNs) such as long short-term memory (LSTM). Later variations have been widely adopted for training large language models (LLM) on large (language) datasets. The modern version of the transformer was proposed in the 2017 paper " Attention Is All You Need" by researchers at Google. Transformers were first developed as an improvement ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Deceptive Alignment
Deception is the act of convincing of one or many recipients of untrue information. The person creating the deception knows it to be false while the receiver of the information does not. It is often done for personal gain or advantage. Deceit and dishonesty can also form grounds for civil litigation in tort, or contract law (where it is known as misrepresentation or fraudulent misrepresentation if deliberate), or give rise to criminal prosecution for fraud. Types Communication The Interpersonal Deception Theory explores the interrelation between communicative context and sender and receiver cognitions and behaviors in deceptive exchanges. Some forms of deception include: * Lies: making up information or giving information that is the opposite or very different from the truth. * Equivocations: making an indirect, ambiguous, or contradictory statement. * Concealments: omitting information that is important or relevant to the given context, or engaging in behavior that hel ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Instrumental Convergence
Instrumental convergence is the hypothetical tendency for most sufficiently intelligent, goal-directed beings (human and nonhuman) to pursue similar sub-goals, even if their ultimate goals are quite different. More precisely, agents (beings with agency) may pursue instrumental goals—goals which are made in pursuit of some particular end, but are not the end goals themselves—without ceasing, provided that their ultimate (intrinsic) goals may never be fully satisfied. Instrumental convergence posits that an intelligent agent with seemingly harmless but unbounded goals can act in surprisingly harmful ways. For example, a computer with the sole, unconstrained goal of solving a complex mathematics problem like the Riemann hypothesis could attempt to turn the entire Earth into one giant computer to increase its computational power so that it can succeed in its calculations. Proposed basic AI drives include utility function or goal-content integrity, self-protection, freedom from ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Value Alignment
Value or values may refer to: Ethics and social sciences * Value (ethics), concept which may be construed as treating actions themselves as abstract objects, associating value to them ** Axiology, interdisciplinary study of values, including ethical values * Social imaginary, set of morals, institutions, laws, and symbols common to a particular social group * Religious values, beliefs and practices which a religious adherent partakes in Economics * Value (economics), a measure of the benefit that may be gained from goods or service ** Theory of value (economics), the study of the concept of economic value ** Value (marketing), the difference between a customer's evaluation of benefits and costs ** Value investing, an investment paradigm * Values (heritage), the measure by which the cultural significance of heritage items is assessed * Present value, value of an expected income stream as of the date of valuation * Present value of benefits, discounted sum of a stream of bene ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
