Can not remove diamond from passive skill tree

I have a controlled metamorphosis diamond that I use to allocate Eldritch Battery on my skill tree. I play on Oracle Druid and have the entwined realities skill equipped. I have a notable passive conduit that will not refund even though all other passive skill points are not chosen. It still will not allow me to refund the notable and remove my diamond. Please help because this is ruining my build. Thank you

It could help a lot if you post a Path of Building for your character.