October challenge problem CHEFTAK is a mess

gorre_morre · October 12, 2018, 7:43pm

As most of you have probably seen is that no one has been able to get an accepted solution to the challenge problem. This might be because of “Sat Oct 06 , 19:49 The checker for CHEFTAK is incorrect. It will be repaired soon and all submissions will be rejudged, sorry for the inconvenience.”. Maybe it still hasn’t been fixed, but I think that the challenge itself also have some big issues.

Issue 1. Constraints

The problem states 0 \le Z \le 20, but the test generator will with probability roughly 1/5 violate these constraints.

Issue 2. Floating point numbers and rounding

The problem states that “We are rounding all the resulting numbers to 8 significant digits”, but I question what this means. For example what is the smallest amount of coins I can buy? I need much more details about the rounding than just “rounding all the resulting numbers to 8 significant digits”.

Issue 3. Scoring

The problem description isn’t entirely clear about what the total score is. It says that the score for each test case is c-10. It also says that non-positive score will result in WA, whether this refers to the total score or separately for individual test case isn’t clear.

Issue 4. Rand

In the test generation is says “Assume that the function rand(l, r) generates a uniformly random number between l and r (both inclusive).” My guess is that this is discrete uniform (integers), but that isn’t clear.

I think that all of these issues need to be addressed, and I suggest the following fixes:

Fix for issue 1

If during the test generation Z ever becomes <0 or >20, then restart the whole test generator. This allows for roughly the same distribution while keeping 0 \le Z \le 20.

Fix for issue 2

Truncate user input to 8 digits of precision. So if the user tries to buy 9.123456789, it will be interpreted as 9.12345678. Other than this do every calculation exact without ever using floating point numbers. With this there will never be any problems concerning rounding, while still behaving essentially the same as before.

Fix for issue 3

Let the total score be the sum of scores over all the test cases and define AC as having a non-negative score on all test cases.

Fix for issue 4

Just be clear about rand being discrete.

With these issues fixed I think this would be a really interesting challenge problem, but as it stands it is unusable.

EDIT: 18/10/2018

The contest has now been extended twice. There was an update on the 16/10/2018 making some fixes for the problem statement (Still missing fixes for issue 2 and some minor errors in the description). The update also stated that “CHEFTAK has been fixed now.” This is in no way true. This time around you can submit solutions and get an “accepted answer” but nothing really works at all. From looking at the scoreboard and reading the forum I understand the following:

Input format is wrong

There are many many solutions with NZEC. From https://discuss.codechef.com/questions/137534/more-issues-with-oct18-challenge-problem-cheftak I understand that this is probably because of the input given not being the same as the one described in the problem description.

Accepted answers aren’t really accepted

Looking at the highscore shows that everyone that got “accepted” only got 0 points. From https://discuss.codechef.com/questions/137534/more-issues-with-oct18-challenge-problem-cheftak, it seems like even printing invalid output like -1 will give accepted answer.

The internal errors

It is possible to get “internal error”. I’m not entirely sure whats causing this. Maybe a bad output can create an internal error or maybe everything is just buggy.

So nothing is working at the moment! I do not understand how the update states “CHEFTAK has been fixed now.”. I understand that there are other competitions around the corner, but this still isn’t acceptable. This is all just a big big mess!

EDIT2 19/10/2018:

User algmyr OCT18 challenge CHEFTAK's testcases confirmed to be completely broken - general - CodeChef Discuss found out the problem with the input. For some reason Y and Z are both identically equal to “0.000000”, this might be because of an undefined behavior in C++ when using printf("%f",x) with x being an int. With gcc it simply prints “0.000000”.

vijju123 · October 13, 2018, 1:22am

The issue with judge should be fixed very soon. Contest will get extended. @admin apologizes for the issue in the judge.

mgch · October 13, 2018, 1:24am

Hi @gorre_morre,

Thanks for your mention, yes your assumptions right:

I’m going to bound Z between 5 and 15, cause when Z is near to the 0 happens unpredictable things. There is also a possibility of mirror reflection if Z becomes more than 15 or less than 5.
Yes, actually, it’s what we really need.
Yes, we’ll do it.
Integer random.

The changes will be produced within 6 hours,

There will be 2 more changes:

We’ll provide 50% tests as public.
And the deadline of submissions for challenge will be extended for 3-5 days.

Thanks! And sorry, there were really strange problems with checker and the system.

vijju123 · October 13, 2018, 1:30am

Look whose back! Missed ya

algmyr · October 13, 2018, 1:34am

I don’t think increasing the ratio of public tests is a good thing. It makes it worthwhile to just optimize for the public cases, which is quite bad for problems with a lot of randomness. Essentially it encourages people to send in a lot of solutions just to optimize the public part of the score, rather than making us try to cope with the randomness of the task. As long as we have correct test generation code for offline testing, keeping the same ratio seems reasonable to me.

mgch · October 13, 2018, 1:43am

@algmyr Solutions can fail on the final set of test cases with very high probability and when you have the negative score on one hidden test case you have only WA.
Maybe better to use multi-testcases for each test case((5 or 10) x 10000) and the score is the sum over them and the same ratio? But with multi-tests it has not much sense.

mgch · October 13, 2018, 1:47am

@vijju123 lol

vijju123 · October 13, 2018, 1:59am

@mgch lolol

gorre_morre · October 13, 2018, 2:01am

@mgch
Sounds great. I really like the idea of mirroring, didn’t think of that. As for the test cases, I’d like to keep most of them hidden to discourage optimizing only for the public test data.

I also like the idea of having 5 \times 10000 test cases as this allows for taking risks while not having to worry to much about getting a WA because of a single bad run. Another good thing about having many test cases is that it rewards general solutions rather than just “lucky” solutions.

jagreetdg · October 13, 2018, 10:04am

6 hours passed, Assuming the checker is successfully repaired, getting this verdict :

internal error occurred in the system

Can anyone explain what is this supposed to mean ??

algmyr · October 13, 2018, 5:44pm

It’s not yet fixed

gorre_morre · October 17, 2018, 1:32am

@mgch Looked through the updated problem, two things, it mentions Z being an integer, also Issue 2. Floating point numbers and rounding still hasn’t been fixed.

vijju123 · October 17, 2018, 9:57am

Forwarded to@ admin to forward to @mgch

ranadev · October 17, 2018, 10:21am

Also the judge has been showing an internal error when I tried to submit few solutions

Ref: https://www.codechef.com/OCT18A/status/CHEFTAK,ranadev

byomkeshbakshy · October 17, 2018, 7:57pm

Now it’s showing “internal error occured in the system!” what’s this now? if you can’t fix it why you are again and again extending the contest?

hbotondi · October 18, 2018, 6:51pm

I still got
“ internal error occurred in the system” (I know it accepted for someone but not for me.) Any advice? (I can submit other tasks but not this)