python - TensorFlow: Compute Hessian matrix (and higher order derivatives) -

i able compute higher order derivatives loss function. @ least able compute hessian matrix. @ moment computing numerical approximation hessian more expensive, , more importantly, far understand, inaccurate if matrix ill-conditioned (with large condition number).

theano implements through symbolic looping, see here, tensorflow not seem support symbolic control flow yet, see here. similar issue has been raised on tf github page, see here, looks nobody has followed on issue while.

is aware of more recent developments or ways compute higher order derivatives (symbolically) in tensorflow?

well, can , little effort, compute hessian matrix!

suppose have 2 variables :

x = tf.variable(np.random.random_sample(), dtype=tf.float32) y = tf.variable(np.random.random_sample(), dtype=tf.float32) 

and function defined using these 2 variables:

f = tf.pow(x, cons(2)) + cons(2) * x * y + cons(3) * tf.pow(y, cons(2)) + cons(4) * x + cons(5) * y + cons(6) 

where:

def cons(x):     return tf.constant(x, dtype=tf.float32) 

so in algebraic terms, function

now define method compute hessian:

def compute_hessian(fn, vars):     mat = []     v1 in vars:         temp = []         v2 in vars:             # computing derivative twice, first w.r.t v2 , w.r.t v1             temp.append(tf.gradients(tf.gradients(f, v2)[0], v1)[0])         temp = [cons(0) if t == none else t t in temp] # tensorflow returns none when there no gradient, replace none 0         temp = tf.pack(temp)         mat.append(temp)     mat = tf.pack(mat)     return mat 

and call with:

# arg1: our defined function, arg2: list of tf variables associated function hessian = compute_hessian(f, [x, y]) 

now grab tensorflow session, initialize variables, , run hessian :

sess = tf.session() sess.run(tf.initialize_all_variables()) print sess.run(hessian) 

note: since function used quadratic in nature (and differentiating twice), hessian returned have constant values irrespective of variables.

the output :

[[ 2.  2.] [ 2.  6.]] 

p.s.: used same example used here .