Augmented Lagrangian method

Consider solving the following constrained optimization problem:

${\displaystyle \min f(\mathbf {x} )}$

subject to

${\displaystyle c_{i}(\mathbf {x} )=0~\forall i\in {\mathcal {E}},}$

where ${\displaystyle {\mathcal {E}}}$ denotes the indices for equality constraints. This problem can be solved as a series of unconstrained minimization problems. For reference, we first list the kth step of the penalty method approach:

${\displaystyle \min \Phi _{k}(\mathbf {x} )=f(\mathbf {x} )+\mu _{k}~\sum _{i\in {\mathcal {E}}}~c_{i}(\mathbf {x} )^{2}.}$

The penalty method solves this problem, then at the next iteration it re-solves the problem using a larger value of ${\displaystyle \mu _{k}}$ and using the old solution as the initial guess or "warm start".

The augmented Lagrangian method uses the following unconstrained objective:

${\displaystyle \min \Phi _{k}(\mathbf {x} )=f(\mathbf {x} )+{\frac {\mu _{k}}{2}}~\sum _{i\in {\mathcal {E}}}~c_{i}(\mathbf {x} )^{2}+\sum _{i\in {\mathcal {E}}}~\lambda _{i}c_{i}(\mathbf {x} )}$

and after each iteration, in addition to updating ${\displaystyle \mu _{k}}$, the variable ${\displaystyle \lambda }$ is also updated according to the rule

${\displaystyle \lambda _{i}\leftarrow \lambda _{i}+\mu _{k}c_{i}(\mathbf {x} _{k})}$

where ${\displaystyle \mathbf {x} _{k}}$ is the solution to the unconstrained problem at the kth step (i.e. ${\displaystyle \mathbf {x} _{k}={\text{argmin}}\Phi _{k}(\mathbf {x} )}$).

The variable ${\displaystyle \lambda }$ is an estimate of the Lagrange multiplier, and the accuracy of this estimate improves at every step. The major advantage of the method is that unlike the penalty method, it is not necessary to take ${\displaystyle \mu \rightarrow \infty }$ in order to solve the original constrained problem. Because of the presence of the Lagrange multiplier term, ${\displaystyle \mu }$ can stay much smaller, and thus avoiding ill-conditioning.[6] Nevertheless, it is common in practical implementations to project multipliers estimates in a large bounded set (safeguards) which avoids numerical instabilities and leads to strong theoretical convergence.[5]

The method can be extended to handle inequality constraints. For a discussion of practical improvements, see refs.[6][5]

The alternating direction method of multipliers (ADMM) is a variant of the augmented Lagrangian scheme that uses partial updates for the dual variables. This method is often applied to solve problems such as,

${\displaystyle \min _{x}f(x)+g(x).}$

This is equivalent to the constrained problem,

${\displaystyle \min _{x,y}f(x)+g(y),\quad {\text{subject to}}\quad x=y.}$

Though this change may seem trivial, the problem can now be attacked using methods of constrained optimization (in particular, the augmented Lagrangian method), and the objective function is separable in x and y. The dual update requires solving a proximity function in x and y at the same time; the ADMM technique allows this problem to be solved approximately by first solving for x with y fixed and then solving for y with x fixed. Rather than iterate until convergence (like the Jacobi method), the algorithm proceeds directly to updating the dual variable and then repeating the process. This is not equivalent to the exact minimization, but the method still converges to the correct solution under some assumptions. Because of this approximation, the algorithm is distinct from the pure augmented Lagrangian method.

The ADMM can be viewed as an application of the Douglas-Rachford splitting algorithm, and the Douglas-Rachford algorithm is in turn an instance of the Proximal point algorithm; details can be found in ref.[7] There are several modern software packages, including YALL1[8] (2009), SpaRSA[9] (2009) and SALSA[10] (2009), which solve Basis pursuit and variants and use the ADMM. There are also packages which use the ADMM to solve more general problems, some of which can exploit multiple computing cores (e.g., SNAPVX[11] (2015), parADMM[12] (2016)).

Stochastic optimization considers the problem of minimizing a loss function with access to noisy samples of the (gradient of the) function. The goal is to have an estimate of the optimal parameter (minimizer) per new sample. With some modifications, ADMM can be used for stochastic optimization. In a stochastic setting, only noisy samples of a gradient are accessible, so an inexact approximation of the Lagrangian is used:

${\displaystyle {\hat {\mathcal {L}}}_{\rho ,k}=f_{1}(x_{k})+\langle \nabla f(x_{k},\zeta _{k+1}),x\rangle +g(y)-z^{T}(Ax+By-c)+{\frac {\rho }{2}}\Vert Ax+By-c\Vert ^{2}+{\frac {\Vert x-x_{k}\Vert ^{2}}{2\eta _{k+1}}},}$

where ${\displaystyle \eta _{k+1}}$ is a time-varying step size.[13]

ADMM has been applied to solve regularized problems, where the function optimization and regularization can be carried out locally and then coordinated globally via constraints.[14][15][16][17]

Regularized optimization problems are especially relevant in the high-dimensional regime as regularization is a natural mechanism to overcome ill-posedness and to encourage parsimony in the optimal solution (e.g., sparsity and low rank). ADMM's effectiveness for solving regularized problems may mean it could be useful for solving hide-dimensional stochastic optimization problems.[18]

• Sequential quadratic programming
• Sequential linear programming
• Sequential linear-quadratic programming

Open source and non-free/commercial implementations of the augmented Lagrangian method:

• Accord.NET (C# implementation of augmented Lagrangian optimizer)
• ALGLIB (C# and C++ implementations of preconditioned augmented Lagrangian solver)
• PENNON (GPL 3, commercial license available)
• LANCELOT (free "internal use" license, paid commercial options)
• MINOS (also uses an augmented Lagrangian method for some types of problems).
• The code for Apache 2.0 licensed REASON is available online.[19]
• ALGENCAN (Fortran implementation of augmented Lagrangian method with safeguards). Available online.[20]
• NLOPT (C++ implementation of augmented Lagrangian optimizer, accessible from different programming languages[21][22])[23]
• PyProximal (Python implementation of augmented Lagrangian method).[24]

• Barrier function
• Interior-point method
• Lagrange multiplier
• Penalty method

• Bertsekas, Dimitri P. (1999), Nonlinear Programming (2nd ed.), Belmont, Mass: Athena Scientific, ISBN 978-1-886529-00-7
• Birgin, E. G.; Martínez, J. M. (2014), Practical Augmented Lagrangian Methods for Constrained Optimization, Philadelphia: Society for Industrial and Applied Mathematics, doi:10.1137/1.9781611973365, ISBN 978-1-611973-35-8
• Nocedal, Jorge; Wright, Stephen J. (2006), Numerical Optimization (2nd ed.), Berlin, New York: Springer-Verlag, ISBN 978-0-387-30303-1