I think your approach to solve the problem is wrong, instead of this you can solve the problem by taking a circle and finding how many points lie exactly in between them. if the two given points are on diametrically opposite points then using mathematics it can be proved that for all the points in between it will form a right angled triangle. otherwise for all other cases the number of minimum number of points lie in between them from clockwise and anticlockwise direction is our solution.