Is there formula or algorithm for that? I can easily compare 2 seven card set but, is there formula for exact rank of my set? I am asking only about full 7 card sets.
I'm not going to lay out the full mathematics, but here is a start that should get you started in the right direction.
First, only the best 5 card hand counts, so find that hand and discard the two cards not in that hand. This should be fairly easy to do so I am not going to cover the algorithm for doing this.
Next, if I understand the question you want to obtain a numerical ranking such that you can compare two hands and see which one is better by doing a simple integer comparison. So you want a function which will return a number
n between 1 and 380204032, where 1-4 are the royal flushes, and 380204032 is one of the instances of 7-5-4-3-2 offsuit. (Note: I'm assuming your math is correct that there are 52^5 possible 5-card hands, as if this is wrong it doesn't really affect my answer, only the number range that would be returned. )
Now in general, the approach is to realize that all royal flushes will take the first spots, followed by straight flushes below ace high, followed by quads, etc. I am not going to go into all the math, but we already know that hands 1-4 will be the four possible royal flushes. Wikipedia gives the number of distinct hands of each rank. There are 9 possible straight flushes (excluding royal flushes) in each suit, for a total of 36 hands, so if the hand you are considering is a straight flush you already know that
36+4 <= n <= 1+4. You then only have to determine which value it is. For a straight flush this is easy, just look at the highest card in the hand, find the difference between it and 13 (where J=11, Q=12, K=13), multiply that by 4, and add 5 (the starting point for straight flushes). For example, a king-high straight flush would be (13-13)*4+5 = 5. At the other end, a steel wheel would be (13-5)*4+5=37. Note that by our way of calculating all tying hands have the same number and we simply never return some numbers, so all four steel wheels return 37, instead of 37, 38, 39, and 40.
Next, create a similar algorithm for all other hand rankings. For quads it will be a little bit more complicated, because you first have to find the ranking of the actual quad, and then consider the kicker, but it is not too much work. For lower hands than quads it does get a bit more complicated, so when you get down to high card you probably have the most work to do. Still, you should be able to get it done in a reasonable amount of time if you partition the problem carefully.
Once you have the algorithm for each type of hand, you merely need to determine what type of hand you have and call the appropriate algorithm for that hand to get the numerical ranking.
Now there is still a possible issue with this approach, because considering all 5 card hands you have a lot of ties. So you will still need to do a bit of work at the end to find all hands above you that tie yours, unless your algorithm for each hand ranks all equal hands with the same number like the straight flush example above. This should not take a lot of time, simply set
i=n-1 and check if hand
i-1 ties hand
n. If so, then, decrement i and repeat the loop. When the loop is broken you, take
i0 is the smallest value which tied