Keras is part of the problem for me. It is rather rigid and it's hard to get around. Works super well for the regular use case. On the other hand, when you want to start doing custom stuff, it's hell.