how to make a formula to solve the project euler problem 577 this link will be helpful I guess.I drew it manually and you need to consider sqrt(3) also has one of the side, see second image.
So, according to me number of hexagons will be equal to number of hexagons with side of length one which is equal sum of integers from 1 to n-2 plus number of hexagons with side of length two and so on. May be this much could help you to further figure out the formula.

I gave my best, even I am trying to figure out the formula. Let me know if you get it. Thank you.
alt text

I managed to find the other hexagon:- (with black bold lines in the diagram)

alt text


i also used same approach but how H(20)=966
n max side of hexagon
3 1 unit [(n-2)+(n-3)+(n-4)…1]
6 2 unit 2[(n-5)+(n-6)+…1]
9 3 unit [(n-8)+(n-9)+…1]
12 4 unit 2[(n-11)+(n-12)+…1]
so if n=12 then formula may be
so am i right or not