Amateur Authors, Lazy Testers and Jan Challange - 2018

general
, , , , ,
1

Hello Everyone!

This is my first post on CodeChef. Before anything else, I would like to request the authors of January Challenge not to take my words personally or as offense. Please go through the entire post (criticism is not always negative) because everyone who is professional now was amateur in the past.

Contest Link : JAN18

This month challenge has problems from varying topics including Suffix Structure, Mobius Algebra, Meet in-the Middle, Sum over Subset, Dynamic Programming and few ad-hoc. I really enjoyed the challenge specially because some of the topics being my favourite. Whenever I get a chance, I always tell juniors to participate in long challenges because one can learn a lot from them.It is clear to see why that is true. You have enough time to read new theories, implement, optimize, learn and also get some Laddus. I started my journey with OCT12, solving just two problems, and since then I have been taking part in long challenges - latest being JAN18.

Can contest problems can have issues? Yes, absolutely. If such issues keep happening, will the quality of such reputed contest remain intact ? I think no. January challenge has few issues that need to be addressed so that it do not repeat in future. Many have already discussed about them in other posts, I will summarize them, add my own thoughts and suggest some ideas. I request everyone else to take part by posting issues and constructive ideas to eliminate them.

  • Gradient of difficulty : Fifth most solved problem see more than 2.5K submissions and next problems sees only 141! Contest admin should take help from other member of problem setting panel to decide on the difficulty. Because many times what seems easy to you, may not be that easy to others and vice versa. Also difficulty can vary according to topic. For example, say I like expected value problems and I can solve them quickly most of the time. However, an ad-hoc / greedy problem which is easier than the expected value problem may ask me for some effort. Whereas, for others who are new into CP, may find the ad-hoc/greedy problem easier because of unfamiliarity with expected values.

  • Lazy Tester : A tester is supposed to write the correct (expected solution) and make sure that it passes on the test data. This is why you are called a tester - because you tested that a correct solution passes. Is it ? Absolutely No. That is just a small part of it. You job is to make sure that test data is strong enough. Write bruteforce, sub-optimal and some heuristic solutions and make sure none of them passes. Suggest corner cases to the author. Validate that constraints are followed and so on. I am not sure, whether a tester is paid for long challenges, but if you are, understand why you are paid. Participants also solve the same problems but they are not paid, so why should you be paid to do the same ?

  • Amatuer Authors : It is good to see so many new problem setters in long challenges - specially young Indian setters. You get to learn a lot when you work with experienced problem setters. At the same time, the experienced setters of the panel and specially the contest admin has responsibility to guide the newcomers. Believe me, a newcomer may be very awesome, bright and what not but there is something in experience that needs to be passed on. Familiarity with the new platform, Time Limit and may be other Platform dependent things are among must that needs to be passed on. Apart from that and most importantly, they need help in preparing strong testdata. They may try their best and claim that data is strong enough. But it is the duty of contest admin and tester to really verify it. In most cases claim will not be true. So, please help them.

  • Irresponsible Authors : If you author a problem, you need to prepare test data, test a valid solution. And submit it for contest. Is this all ? Absolutely No. Once the contest starts. Keep looking into (accepted) solutions as and when they come. Dig into it and see whether it is optimal, sub-optimal, expected, better, correct or anything else. If you spot a solution which should not pass, discuss with the admin and update the test data, put on rejudge. Also, you can learn a lot from the codes. People will submit many innovative heuristics and other kind of correct solutions which you might not have thought. Feel awesome :smiley: Basically, you own that problem - it is upto you to maintain the quality, don’t be irresponsible.

  • KILLKTH Specific : This problem was awesome. However, author has been utterly negligent. I hope, he learnt from it. Looking forward to see more well prepared and beautiful problem from him. Test data contains only small queries. Look My Comment Here and here. Any kind of solution is passing. This is not acceptable even for amatuer author. Advice - when setting problems on platform such as CodeChef where you can not have many input files - go with multiple T testcases, unlike in this problem. And ensure that overall input file is within reasonable limit. Having a single testcase is not suitable at all for CodeChef. As a author write intuitive sub-optimal and make sure it does not pass.

  • XYHUMOQ Specific : Almost - bruteforce solution passes. My Almost Bruteforce Solution. Divide the string into two halves. For each possible flips on first half (at-most 2^15) find out number of subsequences of the form 1010…0 and 1010…1. Do same for second half. Now try to combine each from first to each from second. Total comparison 2^30. Which should time out for any testcase with string length 32 and k = 1 (or any other for which answer is NO). I had many optimizations on mind, but I submitted it without any just in case it passes. I generally do not do this, but did because attempting on the last day. As a author, you should think what can be a worst case. For this problem, when answer is No can constitute as Worst Case for solutions which are exhaustively searching for flip sequence. So you need to beat that.

  • CHEFPALS Challenge Problem : I have not attempted this problem. But looking at the comments of others, it is clear that this problem had issues initially. Even after they fixed, there are some issues. Three people got 100 points. And the next best one and others got very very few points compared to them. I have seen such marking scheme first time in long challenges. I should not write much about it as I have not attempted yet.

Also a note to participants, if you notice that testdata is weak, please inform the author / codechef about it. If the problem is detected earlier, it can be fixed. I generally would do, this time I attempted them on the last day hence could not.

Apart from all this, what do you guys think about partial score problems in long challenge ? I feel like there should be cap of max 20 percent for solving all/any sub-problems. At-least 80 percent should be reserved for the main task. Your opinion, with reason ? I will explain mine after a while.

Repeating, do take part in Long Challenges, they are awesome !

Thanks for reading!

40 Likes
2

Thanks a lot for the feedback :slight_smile: Much appreciated. I have sent the feedback to our admin, Hasan Jaddouh (@kingofnumbers). We, both, will be working on addressing these and keep in mind for future contests.

Specific points related to problems will be discussed separately in that thread. Here, I would only like to mention regarding the difficulty of first 5 problems being easy. Actually, we intend to make the first 5 problems bit easier than usual. The idea is to make it more friendly to new people. Additionally, we have subtasks in all the problems so people can try solving some subtasks in the harder problems. So, there might be such scenarios where the number of people will full solutions go drastically low when moving from 5 problems solved to 6 problems solved. However, due to subtasks, the effect won’t be as drastic as what happened in the Jan contest.

Thanks for the feedback again :slight_smile:
Happy programming !!

10 Likes
3

@admin : I feel that having 5 very easy problem and 4 problems with very high difficulty is not good, even if you want to target newcomers. I personally am never satisfied with solving a challenge partially. Many times I just don’t attempt it, if I don’t have solution for the original task. This may not be true for everyone, but I feel most people won’t be satisfied with partially solving a problem. In the hindsight, you always think how to solve the main problem. Also, as an author I think problem originates in its original version. So solving a sub-task may not even be close to solving the original version. Or may be do a poll to actually understand how most of people thinks ?

That means 5 very easy problem plus 4 hard problems - there is little scope to actually see the improvement for most people. You solve 5 problems for an year and then may be you are able to solve 6 problems. So in this one year, you can not really visualize the improvement.

So have problems with difficulty gradient as smooth as possible. Also have partial scoring on the problem so everyone attempts and score some points. Because the same newcomers you targeted today, may not participate tomorrow because they can not see the improvement. In the long run, technically you will have huge participation, but retention will decline.

Also, if there are other contests for beginners - Lunch Time etc. If there is need, have some other contests. Lowering the quality of an already reputed contest to fit in newcomers, would never be my suggestion.

13 Likes
4

I don’t think you have to make first 5 easy. The first 2-3 are for newcomers, and other 2-3 should be for beginners/lower-intermediate coders.

Making first 5 easy, helps newcomers, as they can solve 4-5 instead of 2-3, but it also ruins the balance for beginners/intermediates, especially with 6th being harder than usual. There was no border between these people, and this can mess up ratings.

4 Likes
5

Also, authors please make editorials when you submit questions. So that they can be posted asap after the contest. This will make contests better.

7 Likes
6

I disagree as well with making first five easy. Steepness in submissions is never good in my opinion. It means lots of ties, and more “discrete” differentiation of participants, which would else be more spaced, continuous.(i mean, we really do not want 1 block of participants having rank 200, and then next 1000 having rank 201. Not good.).

There is a limit upto which subtask of 4 problems can do. And if we begin differentiating by 0.1 point of challenge problem- that will lead to dissatisfaction.

3 Likes
7

This, is completely correct. I agree with you, especially last 2 paragraphs. My opinion is best voiced by this line of yours-

Lowering the quality of an already reputed contest to fit in newcomers, would never be my suggestion.
2 Likes
8

@admin For KILLKTH problem brute force approach in C++ gave only 5 points and in python it gave 20 points. If I had used python then my rank would have been 226 instead of 289. This should not happen, please set appropriate time limit and test cases.

6 Likes
9

The idea of having first 5 easy questions and the rest 4 hard questions is of no practical benefit.

Sure beginners would be able to solve more questions at first, but after a few months they are going to get stuck for a long while when they reach upto 5th problem, which shouldn’t take long.

The path from beginner to intermediate (solving 5th of Jan long) happens relatively quickly. But the next part, going from intermediate to advanced is a very lengthy process.

The most motivating part about long challenges is that you can see your progress. Each time you solve more problems than your previous best, it feels like a milestone and makes you feel like you can keep improving, which is a feeling that would be deprived to all of the intermediate coders.

The difficulty levels should increase in a linear way so that everyone can feel that they are improvising.

If Codechef has enough budget and really want to improvise beginners without affecting the intermediate coders, they might as well start a parallel div 2 long contest with 7th problem of div1 being the 10th problem for div2 or any other reasonable equation.

2 Likes
10

I think there is a confusion about the interpretation of my statement. By making the first 5 problems easier, I meant the following.

Sometimes, even cakewalk problem becomes slightly difficult than intended and it becomes of difficulty like cakewalk-simple. For the first five problems (specifically first 4), if we think the problem is difficult than the desired category, then we will try to make sure that it fits the category, i.e., we will make an error on the categorization of first 4 problems on the easier side rather than on making the problems a bit difficult side.

2 Likes
11

There is a confusion about the interpretation of my statement. I have clarified it up in the above comment. Actually, I meant that we will err on the easier side than on the harder side for the first 5 problems (specifically, it will be more focused on the first four problems).

12

So this is the reason for my first five AC XD

1 Like
13

One of the reasons behind problem CHEFPALS having this steep change (100, 100, 100, ~67, ~10, etc) is that it is a minimization problem. Maximization problems have a smoother gradient usually.

The better the maximum score is in a minimzation problem like this one, the more difficult it is for the others to keep up. Say, if the best score is 10, you would need to get at most 20 to get 50 pts., i.e. within an absolute difference of only 10, which is negligible and means that this submission is almost as good as the top one, but got 50 pts. less.

On the other hand, maximization problems offer a lot more chances. If the top score is 1,000,000 (i.e. 1 million), then to get 50 pts. you need to score at most 500,000 - i.e. within an absolute difference of 500,000 - which is much more manageable. What I mean is that score in points increases linearly with score of submissions, which is not the case in a minimization problem.

4 Likes
14

This makes sense. Now I am interested to know if the scoring function can be maped from exponential into linear type function. I am not saying it is needed, but if can be done, scoreboard will look better. So we won’t see 100, 99.9997 … 99.996 etc. Neither would we see scores like this month.

15

I think that it was quite unfair to award points for wrong brute force solutions for small subtasks or large subtasks.This has happened to me quite a few times and recently in the problem link:https://www.codechef.com/JAN18/problems/PRTITION ,where i was awarded 20 points, may be many others like me for test case :2 8, this was the corner case even though the constraints of Subtask #1 (20 points): sum of N in all test cases ≤ 50, should not allow my solution to pass for 20 points.Actually i think this test case was not included for 20 points subtask. My solution link was :https://www.codechef.com/viewsolution/16891991 .Such issues affect the learning, as new comers may not be able to figure out ,as they will be convinced there answer is correct for small subtasks and create confusions.This is a serious matter of concern.
The quality of problems was very good, but the first one was far too easy and such problem must not given, atleast in long contests.
Happy Coding!

1 Like
16

just to weigh in here, i think the topics to which the question belongs should at least get visible as soon as the contest gets over. the editorials take time to publish, while in the meantime if we can get the topics at least, we can do some analysis about possible approach.

1 Like
17

i agree with the idea that problems have generally steep difficulty level
mostly after 5th.
they shld have gradually increasing difficulty level. This helps us to keep track of our progress. sudden rise in difficulty level demotivates mamy programmers as they think that they are not able to make any progress even after trying so hard

18

This is my first time to post any comment,so Very sorry if my words will hurt u,but this is very necessary to tell u…

this is really very wierd Lazy tester,i got to know that someone posted a solution of Problem-RECTANGL in january long challenge,and it is very obvious that this solution is incorrect but still he/she got full points for this incorrect solution,Now,i am not revealing his/her name,but i can show u that incorrect solution by submitting that solution (by me)…

link of incorrect solution : https://www.codechef.com/viewsolution/17081334

take input : 1 2 3 4
and u will get to know everything…

Please don’t be lazy for next time because your laziness can result into losing faith from codechef…

4 Likes
19

I will forward this to @admin. Thank you. :slight_smile:

20

Must forward @vijju123