@admin , @everyone : I believe final test cases for Challenge Question should be different from those used during the contest . The practice of making hundred’s and thousand’s of submission’s to understand test cases at server is becoming very prevalent and is not in the interest of “MAY THE BEST SUBMISSION WIN” .
I sincerely hope Code Chef admin’s will look into this .
Once the question is moved to practice section you can keep both set of test cases or discard any one , as it doesn’t matter if you do over-fitting once the contest is ended .
Things like money , rating , bragging rights are at stake
I think it is not in interest of CodeChef anyway if people are making thousand’s of submission to learn test data since each submission gets run on SPOJ which costs money to Code Chef .
I don’t think contestants who resorted to such strategy can be blamed because currently Code Chef admin’s have neither raised this issue , nor took preventive action .
I also don’t like this practice of making lots of submissions in order to tune some solution parameters so that, in the end, the solution works better on the official test data (although I did use this approach in the past contests, including the most recent one). On the other hand, I also don’t particularly like the idea of having different tests after the contest (like in TC Marathon matches) - there’s something really appealing in having the certainty of knowing your score during the contest (I am talking about the absolute value of the score, not the relative score which is constatly changing).
However, I believe that using interactive challenge problems with the test cases generated at each submission, like those from the March’13 and May’13 long contests, could partially solve the problem, but not completely.
Anyway, I should say that the challenge problems from the past two contests (July and August) kind of encouraged people to make many submissions because they did not explain the test generation process. When a contestant cannot reproduce the type of test cases used during the evaluation on his own computer there’s no other way to test if his/her ideas are good or not other than submitting it. I would like to encourage future challenge problem setters to use test cases whose generation process can be properly described.
@mugurelionut , @djdolls , @admin , @everyone : On the contraray , I believe the test case generation strategy ( whether OPEN or HIDDEN ) should be same for in contest test cases and final test cases . Forcing the setter to explaining test case generation may be okay , but still what’s the problem if we use a different set of data generated using the same test case generation strategy for FINAL RESULTS. Because the problem I am highlighting is independent of whether test case generation strategy is OPEN or HIDDEN . @djdolls : You will still see people trying to fit their submission to test data if the final test cases are not different .
Those were very good challenge problems as well. And I also wonder what if it would suddenly TLE on the new testdata (or maybe suddenly be wrong). Than I would score zero points? Don’t like that idea much, this problem for example I had AC and TLE for the same code, so changing the data may put it even more at risk.
It might be more difficult to hack the input file if it was permuted randomly, so if there are 14 test cases they’d appear in a random order on each submission. That would be fair as everyone would still get the same test cases.
and it can be used for regular problems as well and tricks to find which input your program is failing on are useless (but it will be very bad if there is wrong format in input file)
@samjay : Suppose there are 10 final test case and 5 contest time test cases , then when you submit during contest it is run on all 15 test cases but you are shown score of only 5 test cases while at the end of contest it will change to score of other 10 test cases . You will not a correct answer verdict during contest if your code gave “Wrong Answer” or “TLE” or “RE” or some other problem occured .
I generally agree with the idea of separating final test cases for Challenge problem.
(+1) I agree with the point that, thousand’s of submission to learn test data is super boring.
(+10) Prefer to challenge problems whose generation process can be properly opened.
(+100) I totally agree that separated final test cases should have the same distribution of provisional test cases, though the contests needn’t to be the similar format as TC Marathon matches.
(-1) Interactive challenge problems may partially solve the problem, but it may also increase the number of submissions.
(-1) We may need much longer time to test it out.
(-10) One potential problem : the (final) submission got “Wrong Answer” or “TLE” on some final test cases?
Some ideas :
Test the submission on all cases, but only show scores for the first 10%. And use the later 90% to determine the winner.
How about having a CAPTCHA ( Completely Automated Public Turing test to tell Computers and Humans Apart ) before every challenge problem submission . That would prevent people from using scripts and force them to do whatever experimentation they need to do manually , which they can do only in a limited way in a 10 day contest .
This should solve the problem considerably as last time max number submission of submissions on the challenge problem by a given user was around 5000
Looking forward to a “FAIR” September LONG contest , where i don’t see more than 500 submissions for challenge problem by anyone . Well, that’s my idea of “FAIR” .
This may break things for those who use a command line browser or have built a command line utility to make submissions. It might also be frustrating for users to enter a captcha each time they want to make a submission. We are open to suggestions on this.
hm… I made all my submissions manually (for all the challenge problems so far) and I still ended up with a bit more than 2300 submissions for the August’13 challenge problem (and I did not spend all my time on the challenge problem, as I also have other things to do in my day-to-day life ). With more perseverence and dedication I guess it is possible to reach even around 5000 submissions manually. Nevertheless, a captcha would definitely slow things down a bit (as well as a higher minimum duration between submissions, which is currently 30 seconds).
A 6 minute gap between submissions would mean 14400 / 6 = 2400 maximum submissions
And if people are not using scripts , that mean they make submissions only during day and the time when they are sitting on system ( say about half the day , 12 hours/day ) would mean max 1200 submissions .
Dont’ know if this enough to solve the issue at hand . But could be a good step nevertheless .
@admin : Do people really use command line browser to access CodeChef , is there any evidence to it ???
Not being able to use some utility to make submission just for 1 out of 10 problems should not be a matter of concern .
Similary , frustration should be limited as we ask for CAPTCHA only for CHALLENGE problem and not otherwise .
All my suggestions on this thread relate only to CHALLENGE problem and should apply only to it .
6 minutes is too long to take break… it would be very boring to tackle … @vineetpaliwal no one has as much patience as you have …I think this idea wont work
what can be done is that make strong testcases with largelimits so that they cant be recognised by submissions (assert ) + one can have difern final testcase for final result (which I completely support)
@eagle_eye : Thanks for the feedback . I too feel such things as time limit between submissions , use of CAPTCHA are all just unnecessary workarounds . I still vouch for separate in contest and end contest test files for CHALLENGE problem .
(but it will be very bad if there is wrong format in input file)