In my opinion, the following are a must in the interest of fairness:
a) Test data generation should be made public. This is because my final solution strongly depends on what generation scheme is used.
- Theoretically, I should be able to decide which of my schemes are better,
- For inputs with multiple parameters, often strategies strongly depend of relative distribution of parameters and I must know them before hand.
- Test data can be designed in adversarial fashion for some “good” schemes and If I am not aware of a it, my “good” scheme could actually end up doing worse than a “bad” scheme.
- People can spend time more usefully in cooking up solutions rather than figuring out the test cases. Nobody likes to do it, but people are left with little choice.
b) Final test data should be different from the one used during contests.
- People wont make 1000s of submissions trying to align their strategies with the judge’s test data.
- People can rate their solutions offline and be assured that it is a good enough estimate of the actual score they are going to get.
- The better strategy will win with more probability as no test data specific hacks will work.
If a) and b) are enforced then number of solutions will go down automatically, without need for captcha and all.
We can allow people to mark some 5-10 submissions and each can be run for the final test data(say last 10 submissions). This is because they could have used different schemes that have similar results, then they may want all of them to be used for final testing. Making this number small enough will ensure that people only put solutions with different ideas/schemes, at the same time allowing room / incentive for more creativity.