In current contest, SSTORY and STREETTA were rejudged and it came to be handy as almost half the solution found to be incorrect. Fortunately the admins were able to find that the test cases were weak. But this may not happen always and even incorrect solutions may end up being accepted throughout the contest.
So shouldn’t be there some provision for the users who have got the problem accepted, to check the others solution, just like Hack option in codeforces? Instead of revealing the code other options may be to provide some interface to find the output of some code for some test case. This system would really help in handling the problem of weak cases as I think people would give good time to test other solutions to eliminate any accepted wrong submission.
1 Like
I made similar proposal in the past and I completely agree with you…
But this task is quite difficult to implement
- there are thousands of contestants in the contest so if everyone adds test, there are thousands of test for a problem… I also proposed a solution for this - your test is accepted only if the format is correct and at least one ACed solution fails your solution…
- even with such policy, those thousands of users can make the CodeChef page unavailable, solution here can be, that your rating to add test case have to be above average of all contestants or something like that
When I asked admins, if it’s possible to add test cases in practice I was told, that setter have to confirm those test cases, so I’m afraid that the addition of test cases during a contest is even more difficult…
3 Likes
I think this system will work great for a Short Contest but be very difficult to implement in a Long Contest and also not practically feasible. There have been a total of 63921 submissions (so far) for all problems in March Long. Running user generated test cases on such a large scale will be very expensive and time consuming.
One thing i feel which is possible is for users to generate some good corner cases and submit these to the Setters/Testers through some interface. If the Setters feel that these test cases are good then they can incorporate them in their input files. There could be some reward mechanism to reward people who give good test cases (Karma, Rating Points ,etc ).
All this is fine in theory but i still feel its very difficult in Practice. Right now we can only hope the Setters and Testers have done their job well enough to ensure a smooth contest.
1 Like
Yes, those numbers are huge, recently there are 3500 + 2500 ACed solution for first two most solved solutions, and to rejudge all of those is CPU time consuming for sure… But I think, that in short contests this is even worse - when you take into account all those cheaters here I can imagine, that someone will overload the server, just to disable you to submit… Or someone can add test case just few seconds before contest end and so on (yeah that’s another problem in long contest to be handled too)…
Totally agreed. Lots of problems in implementing such a mechanism. Don’t think its practical here on codechef.
Yeah, its difficult to run user generated test case on all files, but let the user test his test-case on any of the accepted submission. If it fails, that he can submit it to the setters and they can review it. Its not necessary to run user test-case on all files. Moreover for handling many concurrent process, limit the number of test-case testing at a particular time. The user can do this at some other time in case of congestion. Topcoder gives 10 minutes solely for such process. May be by making some constraints such as rating and ranking as betlista says, it could be made possible.