You're right, you'd better not to hinder me in it/ from doing it.

You're right, it would be better not to prevent me from doing it.

You are right, it's better not to detract me from it.

You are right, it's better not to hinder me in this case.

You are right, it's better not to thwart me in this case.