Can I seek sole/legal physical custody after all this?

I have previously arranged for my children to live with their father for approximately 6 months; in the state of Connecticut where he now resides arriving from NYS with children. I requested they come back that following summer and only got the youngest child. He does not want to pay child support was his excuse for not turning the others over. ( We have a NYS court order) I came 19 months later and got the other children and brought them to Texas where I have been residing for a few months. I have expressed to him that they should have pports for travel . ( My family and I are from Canada) He has ignored several written/verbal requests. I even bartered terminating NYS child support just for his signature. No luck there. I am aware he doesn't want to go to court period. Yea, I may have been too nice and my attempts were to be fair for the sake of the kids. Anyhow, I moved forward a long time ago and we cant seem to ever make clear decisions together. ( Isn't that called joint custody?) He has put his responsibilities in the hands of his current girlfriend. What was alarming is she told (just to name a few) my 12 year old that she would steal them if I get them back. My intentions was for the children to get to know their father and keep in touch with family in the area; not to be brainwashed by his girlfriend. Therefore, civil communication wasn't all that effective. My inquiry is where is the jurisdiction? We had a few cases in regards to CS and restraining orders, in NY. None where custody was established. Thank you for your time and cooperation.