|
Sequential Minimal Optimization
Sequential minimal optimization (SMO) is an algorithm for solving the quadratic programming (QP) problem that arises during the training of support-vector machines (SVM). It was invented by John Platt in 1998 at Microsoft Research. SMO is widely used for training support vector machines and is implemented by the popular LIBSVM tool. The publication of the SMO algorithm in 1998 has generated a lot of excitement in the SVM community, as previously available methods for SVM training were much more complex and required expensive third-party QP solvers. Optimization problem Consider a binary classification problem with a dataset (''x''1, ''y''1), ..., (''x''''n'', ''y''''n''), where ''x''''i'' is an input vector and is a binary label corresponding to it. A soft-margin support vector machine is trained by solving a quadratic programming problem, which is expressed in the dual form as follows: :\max_ \sum_^n \alpha_i - \frac12 \sum_^n \sum_^n y_i y_j K(x_i, x_j) \alpha_i \alpha_j, :s ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Optimization Algorithm
Mathematical optimization (alternatively spelled ''optimisation'') or mathematical programming is the selection of a best element, with regard to some criterion, from some set of available alternatives. It is generally divided into two subfields: discrete optimization and continuous optimization. Optimization problems of sorts arise in all quantitative disciplines from computer science and engineering to operations research and economics, and the development of solution methods has been of interest in mathematics for centuries. In the more general approach, an optimization problem consists of maximizing or minimizing a real function by systematically choosing input values from within an allowed set and computing the value of the function. The generalization of optimization theory and techniques to other formulations constitutes a large area of applied mathematics. More generally, optimization includes finding "best available" values of some objective function given a define ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Lagrange Multiplier
In mathematical optimization, the method of Lagrange multipliers is a strategy for finding the local maxima and minima of a function subject to equality constraints (i.e., subject to the condition that one or more equations have to be satisfied exactly by the chosen values of the variables). It is named after the mathematician Joseph-Louis Lagrange. The basic idea is to convert a constrained problem into a form such that the derivative test of an unconstrained problem can still be applied. The relationship between the gradient of the function and gradients of the constraints rather naturally leads to a reformulation of the original problem, known as the Lagrangian function. The method can be summarized as follows: in order to find the maximum or minimum of a function f(x) subjected to the equality constraint g(x) = 0, form the Lagrangian function :\mathcal(x, \lambda) = f(x) + \lambda g(x) and find the stationary points of \mathcal considered as a function of x and the Lagrange mu ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Kernel Perceptron
In machine learning, the kernel perceptron is a variant of the popular perceptron learning algorithm that can learn kernel machines, i.e. non-linear classifiers that employ a kernel function to compute the similarity of unseen samples to training samples. The algorithm was invented in 1964, making it the first kernel classification learner. Preliminaries The perceptron algorithm The perceptron algorithm is an online learning algorithm that operates by a principle called "error-driven learning". It iteratively improves a model by running it on training samples, then updating the model whenever it finds it has made an incorrect classification with respect to a supervised signal. The model learned by the standard perceptron algorithm is a linear binary classifier: a vector of weights (and optionally an intercept term , omitted here for simplicity) that is used to classify a sample vector as class "one" or class "minus one" according to :\hat = \sgn(\mathbf^\top \mathbf) where a ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Bregman Method
The Bregman method is an iterative algorithm to solve certain convex optimization problems involving regularization. The original version is due to Lev M. Bregman, who published it in 1967. The algorithm is a row-action method accessing constraint functions one by one and the method is particularly suited for large optimization problems where constraints can be efficiently enumerated. The algorithm works particularly well for regularizers such as the \ell_1 norm, where it converges very quickly because of an error cancellation effect. Algorithm In order to be able to use the Bregman method, one must frame the problem of interest as finding \min_u J(u) + f(u), where J is a regularizing function such as \ell_1. The ''Bregman distance'' is defined as D^p(u, v) := J(u) - (J(v) + \langle p, u - v\rangle) where p belongs to the subdifferential of J at u (which we denoted \partial J(u)). One performs the iteration u_:= \min_u(\alpha D(u, u_k) + f(u)), with \alpha a constant to be c ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Vladimir Vapnik
Vladimir Naumovich Vapnik (russian: Владимир Наумович Вапник; born 6 December 1936) is one of the main developers of the Vapnik–Chervonenkis theory of statistical learning, and the co-inventor of the support-vector machine method, and support-vector clustering algorithm. Early life and education Vladimir Vapnik was born to a Jewish family in the Soviet Union. He received his master's degree in mathematics from the Uzbek State University, Samarkand, Uzbek SSR in 1958 and Ph.D in statistics at the Institute of Control Sciences, Moscow in 1964. He worked at this institute from 1961 to 1990 and became Head of the Computer Science Research Department. Academic career At the end of 1990, Vladimir Vapnik moved to the USA and joined the Adaptive Systems Research Department at AT&T Bell Labs in Holmdel, New Jersey. While at AT&T, Vapnik and his colleagues did work on the support-vector machine, which he also worked on much earlier before moving to the USA. They de ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Isabelle Guyon
Isabelle Guyon (; born August 15, 1961) is a French-born researcher in machine learning known for her work on support-vector machines, artificial neural networks and bioinformatics. She is a Chair Professor at the University of Paris-Saclay. She is considered to be a pioneer in the field, with her contribution to the support-vector machines with Vladimir Vapnik and Bernhard Boser. Biography After graduating from the French engineering school ESPCI Paris in 1985, she joined the group of Gerard Dreyfus at the Université Pierre-et-Marie-Curie to do a PhD on neural networks architectures and training. Guyon defended her thesis in 1988 and was hired the year after at AT&T Bell Laboratories, first as a post-doc, then as a group leader. She worked at Bell Labs for six years, where she explored several research areas, from neural networks to pattern recognition and computational learning theory, with application to handwriting recognition. She collaborated with Yann LeCun, ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Bernhard Boser
Bernhard is both a given name and a surname. Notable people with the name include: Given name *Bernhard of Saxe-Weimar (1604–1639), Duke of Saxe-Weimar *Bernhard, Prince of Saxe-Meiningen (1901–1984), head of the House of Saxe-Meiningen 1946–1984 *Bernhard, Count of Bylandt (1905–1998), German nobleman, artist, and author *Prince Bernhard of Lippe-Biesterfeld (1911–2004), Prince Consort of Queen Juliana of the Netherlands *Bernhard, Hereditary Prince of Baden (born 1970), German prince *Bernhard Frank (1913–2011), German SS Commander *Bernhard Garside (born 1962), British diplomat *Bernhard Goetzke (1884–1964), German actor *Bernhard Grill (born 1961), one of the developers of MP3 technology *Bernhard Heiliger (1915–1995), German sculptor *Bernhard Langer (born 1957), German golfer *Bernhard Maier (born 1963), German celticist * Bernhard Raimann (born 1997), Austrian American football player *Bernhard Riemann (1826–1866), German mathematician *Bernhard Siebken ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Karush–Kuhn–Tucker Conditions
In mathematical optimization, the Karush–Kuhn–Tucker (KKT) conditions, also known as the Kuhn–Tucker conditions, are first derivative tests (sometimes called first-order necessary conditions) for a solution in nonlinear programming to be optimal, provided that some regularity conditions are satisfied. Allowing inequality constraints, the KKT approach to nonlinear programming generalizes the method of Lagrange multipliers, which allows only equality constraints. Similar to the Lagrange approach, the constrained maximization (minimization) problem is rewritten as a Lagrange function whose optimal point is a saddle point, i.e. a global maximum (minimum) over the domain of the choice variables and a global minimum (maximum) over the multipliers, which is why the Karush–Kuhn–Tucker theorem is sometimes referred to as the saddle-point theorem. The KKT conditions were originally named after Harold W. Kuhn and Albert W. Tucker, who first published the conditions in 1951. L ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Kernel Function
In operator theory, a branch of mathematics, a positive-definite kernel is a generalization of a positive-definite function or a positive-definite matrix. It was first introduced by James Mercer in the early 20th century, in the context of solving integral operator equations. Since then, positive-definite functions and their various analogues and generalizations have arisen in diverse parts of mathematics. They occur naturally in Fourier analysis, probability theory, operator theory, complex function-theory, moment problems, integral equations, boundary-value problems for partial differential equations, machine learning, embedding problem, information theory, and other areas. This article will discuss some of the historical and current developments of the theory of positive-definite kernels, starting with the general idea and properties before considering practical applications. Definition Let \mathcal X be a nonempty set, sometimes referred to as the index set. A symmetri ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Quadratic Programming
Quadratic programming (QP) is the process of solving certain mathematical optimization problems involving quadratic functions. Specifically, one seeks to optimize (minimize or maximize) a multivariate quadratic function subject to linear constraints on the variables. Quadratic programming is a type of nonlinear programming. "Programming" in this context refers to a formal procedure for solving mathematical problems. This usage dates to the 1940s and is not specifically tied to the more recent notion of "computer programming." To avoid confusion, some practitioners prefer the term "optimization" — e.g., "quadratic optimization." Problem formulation The quadratic programming problem with variables and constraints can be formulated as follows. Given: * a real-valued, -dimensional vector , * an -dimensional real symmetric matrix , * an -dimensional real matrix , and * an -dimensional real vector , the objective of quadratic programming is to find an -dimensional vector , that wi ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Dual Problem
In mathematical optimization theory, duality or the duality principle is the principle that optimization problems may be viewed from either of two perspectives, the primal problem or the dual problem. If the primal is a minimization problem then the dual is a maximization problem (and vice versa). Any feasible solution to the primal (minimization) problem is at least as large as any feasible solution to the dual (maximization) problem. Therefore, the solution to the primal is an upper bound to the solution of the dual, and the solution of the dual is a lower bound to the solution of the primal. This fact is called weak duality. In general, the optimal values of the primal and dual problems need not be equal. Their difference is called the duality gap. For convex optimization problems, the duality gap is zero under a constraint qualification condition. This fact is called strong duality. Dual problem Usually the term "dual problem" refers to the ''Lagrangian dual problem'' but other ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Support Vector Machine
In machine learning, support vector machines (SVMs, also support vector networks) are supervised learning models with associated learning algorithms that analyze data for classification and regression analysis. Developed at AT&T Bell Laboratories by Vladimir Vapnik with colleagues (Boser et al., 1992, Guyon et al., 1993, Cortes and Vapnik, 1995, Vapnik et al., 1997) SVMs are one of the most robust prediction methods, being based on statistical learning frameworks or VC theory proposed by Vapnik (1982, 1995) and Chervonenkis (1974). Given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm builds a model that assigns new examples to one category or the other, making it a non- probabilistic binary linear classifier (although methods such as Platt scaling exist to use SVM in a probabilistic classification setting). SVM maps training examples to points in space so as to maximise the width of the gap between the two categories. New ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
