Yes it is a deal breaker because anyone who uses has his primary relationship with pot and not with you. They are loyal to their drug and it is like being a third wheel your own relationship.
As much as I've tried to open my mind. I think anyone that uses it daily or very frequently, indeed, makes pot the priority and its their primary relationship.
You will be the third wheel, unless you fully accept and participate. But I'm not sure where that gets you.