adanet input as indices of other matrices

The X input is a matrix of indices from other matrices how can we resolve it at runtime to the real values?
e.g.:

X = [[1,1],[1,2]]
Y = [0,1]
lookup_col1 = {1:[1,1,1,1,1]}
lookup_col2 = {1:[3,3,3,3,3], 2:[6,6,6,6,6]}

and what we want to train on is:

X_real = [[1,1,1,1,1,3,3,3,3,3],[1,1,1,1,1,6,6,6,6,6]]