Question

In the lectures, we introduced Gradient Descent, an optimization method to find the minimum value of a function. In this problem we try to solve a fairly simple optimization problem: min f(x) = x2 TER That is, finding the minimum value of x2 over the real line. Of course you know it is when x = 0, but this time we do it with gradient descent. Recall that to perform gradient descent, you start at an arbitrary initial point xo, and you update Xt+1 = Xt – 1Vxf(xt), where is the learning rate. Hopefully, after T iterations, XT will be close to the minimum point. (a) (10 pts) Assume xo = 1 and we choose the learning rate to be l = 1. Now suppose a sequence, X1, ... , XT, is obtained through gradient descent algorithm. Prove that for arbitrary T > 0, f(XT) = 1. Hence, the gradient descent fails completely. Can you provide an intuitive explanation as to why? (b) (10 pts] Assume Xo = 1 and \ = 2. Prove that Xt+1 > xt is always true. The gradient descent even increases the function value! (c) (10 pts] What is the reason gradient descent fails to work in the above two cases, even for a simple optimization problem? What can be done to make gradient descent work? (You don't need a perfect solution here. In fact, a lot of research, even today, have been put into improving the stability and efficiency of (stochastic) gradient descent algorithms.)

Answer 1

Solution & According to given data. a) Given 20=1, 9=1; PLX) ==} Ox = 022 Joc = 9x :-) x + 1 = 4-dox Flat) > 0 L above values sub in E90 Ext-CX2X4** (XL) . :94+1 car 11-2F (2+)) 6=0 El Xoti = Xo (1-26 (20) case-Xi =';1-2X/ case-2 t = 1 xt1 =:8 (1-2f (x;)). .: 2 X2 =(-) LI-2X) x=1 = the repeted values as landni ff is will appear for xt It is f (y al and F(-1) =)

- = the four) is always bel slepkakel - greppet ot V , descent : : . Prg.a prg. a a) this according Rigia . It happen's why of is learning rate as to to taken vs boo step being :) This descent : highe and it makes yo too too ahead. b) Given do=1 and 1=2 a = Xo-X (206) = 1-2(2-1) =) = 1 Haz .:X43.39 dr zx, -2 { 2 6:3-2)} :- 3+2 +128. 2 = 9:6 wE+IE 84-2c5f (I+)) : by Solving we get att =-3x4

267x) = AXT+ out 6 flx+1) = lazi123 to ea sub the "XLAH" Et valve =(-30492 f(x++) = 9242 Now Pixi Paxti+1) 1-9 for Rapt sa asasi Rixt) fort =) this always to trure Reaga > Ruxt)