Continuous Assessment Introduction to AI

DISCLAIMER:

All marks are provisional, may be changed upon further scrutiny, and need approval by various boards of examiners. There are given here exclusively in order to give a first indication and without any guarantee.

Since we expect higher achievements from three member teams than from two member teams and the group mark does not reflect this, we add 10 to the group mark to compute the individual marks in a two member team.

Marks are scaled with respect to the contribution of each team member in the range of plus minus 10 marks so that the average of the marks of the group members is the group mark. Special rules apply in the case of minor contributions of a group member.

You can find general comments on the marking at the end of this file.

Marks for Report only Assignments

The report only projects were marked using the criteria:
1. (20%) Interestingness
2. (20%) Thoroughness
3. (10%) Understandability
4. (10%) Style
5. (20%) Appraisal
6. (10%) Literature
7. (10%) Novelty

GRIDVIVACONTRIBPEN-
ALTY
MARK FOR REPORTGR
MARK
TOTALID
1234567
1 323508 No 50 0 18 18 9 8 10 9 3 75 85 323508
592346 50 85 592346
2 329172 No 50 0 12 10 8 7 12 7 1 57 67 329172
326786 50 67 326786
3 625335 No 100 0 10 8 12 5 9 5 2 51 71 625335

Marks for Report plus Implementation Assignments

The marks for the report and the implementation are given following the criteria:
 1. (7%) Idea novel, combination of ideas from literature/worksheet, standard technique.
 2. (3%)   justification for ideas
 3. (3%) Implementation description of main algorithm clear/sort of clear/unclear
 4. (2%)   justification of design decisions given/missing
 5. (5%) Evaluation description of experimental setup (very good, good, mediocre, poor, missing)
 6. (5%)   results and experiments are clear, thorough, poor, missing
 7. (5%)   experimental setup: hold out (2%) or cross validation (5%)
 8. (5%) Assessment synthesis of results (e.g. precision)
 9. (7%)   discussion of results: clear, correct, muddled, missing
10. (8%)   explanation of results: clear, correct, muddled, missing
11. (5%)   possible improvements
12. (5%) References present, details correct, style correct
13. (20%) Performace  
14. (5%) Quality of code appropriate comments
15. (5%)   idiomatic code
16. (10%)   organisation of code
17.   bonus points for particularly interesting approach

GRIDVIVACONTRIBPEN-
ALTY
MARK FOR REPORTSUM MARK FOR PROGRAMSUMGR
MARK
TOTALID
1234567 8910111213 14151617
4 322903 No 33 0 2 1 1 0 2 3 2 3 3 3 2 3 25 0 4 4 7 0 15 40 40 322903
543376 33 40 543376
545198 33 40 545198
5 323104 No 50 0 4 1 2 1 4 3 4 3 4 5 3 2 36 14 2 4 6 0 26 62 72 323104
330383 50 72 330383
6 328274 DONE 33 5 5 2 2 1 4 4 3 2 2 3 1 3 32 20 3 4 6 0 33 60 60 328274
329460 33 60 329460
653612 33 60 653612
7 328905 No 70 0 3 1 2 1 3 2 1 2 3 3 3 2 26 15 5 3 5 0 28 54 68 328905
326099 30 60 326099
8 329017 DONE 50 5 5 1 2 1 3 2 2 3 3 2 3 2 29 15 3 3 6 -4 23 47 57 329017
329831 50 57 329831
9 330319 No 37 5 4 2 1 1 3 4 2 3 3 4 3 2 32 0 5 3 8 0 16 43 45 330319
328143 37 45 328143
325824 26 39 325824
10 330366 No 50 0 6 3 2 1 3 3 2 3 5 4 1 0 33 14 2 2 5 0 23 56 66 330366
326515 50 66 326515
11 330612 No 50 0 3 1 1 1 1 1 1 1 2 2 2 3 19 0 2 2 4 0 8 27 37 330612
328338 50 37 328338
12 330684 No 50 5 3 1 2 0 2 3 1 2 4 2 1 2 23 7 3 3 6 0 19 37 47 330684
587540 50 47 587540
13 514190 DONE 50 0 2 1 1 1 1 0 0 0 1 1 2 2 12 0 0 2 4 2 8 20 30 514190
540475 50 30 540475
14 515297 No 50 0 4 2 2 1 4 3 4 3 4 4 3 3 37 20 2 3 5 0 30 67 77 515297
530442 50 77 530442
15 517206 No 50 0 3 1 2 1 3 3 2 2 3 3 4 3 30 18 3 3 5 0 29 59 69 517206
532440 50 69 532440
16 517571 No 33 0 3 1 2 1 4 3 3 3 3 3 2 2 30 0 3 3 6 0 12 42 42 517571
524982 33 42 524982
558328 33 42 558328
17 521178 No 50 0 3 2 1 1 2 3 1 2 4 3 4 2 28 18 3 3 5 0 29 57 67 521178
543728 50 67 543728
18 522586 DONE 50 5 3 1 2 1 2 2 1 2 2 3 2 2 23 0 2 4 6 0 12 30 40 522586
537037 50 40 537037
19 523904 No 50 0 6 3 3 2 5 4 5 3 6 7 4 3 51 18 2 1 3 0 24 75 85 523904
540640 50 85 540640
20 524732 No 50 0 7 2 2 1 3 3 1 3 6 6 4 2 40 0 3 3 6 3 15 55 65 524732
525148 50 65 525148
21 525308 No 50 0 6 2 3 2 4 4 2 3 4 5 3 3 41 18 3 3 6 0 30 71 81 525308
527219 50 81 527219
22 525502 No 50 0 2 1 1 1 0 0 0 0 0 0 2 1 8 7 0 0 0 0 7 15 25 525502
518833 50 25 518833
23 525574 No 50 0 3 2 2 1 0 1 0 0 2 2 2 0 15 0 2 3 5 0 10 25 35 525574
595371 50 35 595371
24 525716 DONE 50 0 4 2 2 2 4 3 4 3 4 5 3 3 39 20 2 3 6 -10 21 60 70 525716
326044 50 70 326044
25 527539 No 60 0 3 0 1 0 0 0 0 0 0 0 1 3 8 0 3 3 6 0 12 20 32 527539
540516 40 28 540516
26 528972 No 50 0 6 2 2 2 3 3 0 1 2 1 0 4 26 16 4 4 8 0 32 58 68 528972
580571 50 68 580571
27 529053 No 40 0 2 1 3 1 2 3 1 2 2 3 0 1 21 0 1 3 3 0 7 28 32 529053
548495 30 26 548495
542054 30 26 542054
28 529821 No 70 5 6 3 2 1 3 3 1 2 3 4 1 2 31 0 4 4 7 0 15 41 55 529821
536000 30 47 536000
29 530241 No 50 0 3 2 1 1 2 2 1 2 1 1 1 3 20 0 1 2 3 0 6 26 36 530241
514483 50 36 514483
30 530428 DONE 40 0 6 2 2 1 2 1 0 1 1 1 1 4 22 16 4 4 7 0 31 53 57 530428
572582 40 57 572582
522296 20 45 522296
31 531507 No 50 0 2 1 1 1 1 1 0 1 1 1 0 0 10 14 2 3 5 0 24 34 44 531507
533076 50 44 533076
32 532540 No 50 0 6 3 2 2 4 4 2 2 5 4 4 3 41 18 3 3 6 0 30 71 81 532540
523982 50 81 523982
33 533293 No 33 0 2 1 1 1 2 2 2 2 2 2 1 4 22 18 4 3 4 0 29 51 51 533293
559930 33 51 559930
324334 33 51 324334
34 533706 No 33 0 4 2 1 1 2 3 1 1 2 3 2 2 24 14 3 4 8 0 29 53 53 533706
538649 33 53 538649
515175 33 53 515175
35 536510 No 50 0 3 2 2 1 2 2 1 1 3 3 1 2 23 0 3 3 7 0 13 36 46 536510
541494 50 46 541494
36 536645 No 100 5 4 2 2 1 3 3 1 2 3 3 0 2 26 0 4 4 6 0 14 35 55 536645
37 536690 No 100 0 3 1 2 1 2 1 1 1 2 2 0 2 18 18 4 4 8 0 34 52 72 536690
38 536702 No 100 0 3 2 1 1 2 3 1 1 2 1 3 4 24 0 3 3 7 0 13 37 57 536702
39 537907 No 54 0 6 3 2 2 4 4 2 3 5 6 4 4 45 18 4 4 6 0 32 77 88 537907
530995 46 86 530995
40 538023 DONE 33 0 4 2 2 2 3 3 1 2 1 3 3 2 28 19 4 4 7 -5 29 57 57 538023
557586 33 57 557586
549800 33 57 549800
41 538432 DONE 50 0 4 2 2 2 2 3 1 1 2 2 3 4 28 0 4 4 7 2 17 45 55 538432
536483 50 55 536483
42 539808 No 50 0 3 2 1 1 2 2 1 1 3 3 2 3 24 0 3 4 7 0 14 38 48 539808
529279 50 48 529279
43 540631 No 50 0 6 2 2 2 3 3 1 2 2 3 3 3 32 0 3 3 6 0 12 44 54 540631
534033 50 54 534033
44 541537 No 50 0 3 1 2 1 4 4 2 3 4 5 4 3 36 16 3 4 7 0 30 66 76 541537
511770 50 76 511770
45 542571 No 55 0 5 3 2 1 2 3 1 1 3 2 2 0 25 0 3 3 6 0 12 37 48 542571
517100 45 46 517100
46 543718 No 33 0 3 1 2 1 2 2 1 1 2 1 2 1 19 0 3 3 6 0 12 31 31 543718
329659 33 31 329659
548496 33 31 548496
47 544501 No 50 0 6 3 2 1 4 4 2 2 4 4 3 3 38 0 3 4 7 0 14 52 62 544501
542431 50 62 542431
48 544833 DONE 41 0 3 2 3 2 4 4 1 2 5 4 4 5 39 0 2 3 6 0 11 50 55 544833
543375 30 48 543375
328335 29 48 328335
49 549993 No 50 0 3 2 2 1 2 1 1 1 1 2 2 4 22 0 3 3 6 0 12 34 44 549993
530044 50 44 530044
50 550737 DONE 50 10 4 2 2 1 2 2 1 1 1 2 1 3 22 0 3 3 6 0 12 24 34 550737
551450 50 34 551450
51 552536 No 50 0 6 3 3 2 4 4 1 2 3 5 2 3 38 19 3 3 6 2 33 71 81 552536
537123 50 81 537123
52 552974 No 90 0 6 2 2 2 4 4 1 3 3 4 2 2 35 18 3 3 4 0 28 63 81 552974
329392 10 15 329392
53 555361 No 66 0 6 2 2 2 5 5 1 2 5 5 3 3 41 0 3 3 6 0 12 53 66 555361
662476 33 60 662476
54 556708 No 100 0 4 1 2 1 3 3 1 2 2 3 2 3 27 0 3 3 6 0 12 39 59 556708
55 558350 No 50 0 4 3 2 2 4 4 1 2 4 3 1 3 33 16 3 3 6 -5 23 56 66 558350
643441 50 66 643441
56 558502 No 34 0 4 2 2 1 4 3 4 2 3 2 1 2 30 15 3 3 6 0 27 57 58 558502
544839 33 57 544839
580762 33 57 580762
57 560175 DONE 50 0 3 2 3 2 4 4 5 2 3 2 2 3 35 20 3 3 6 -10 22 57 67 560175
536755 50 67 536755
58 562510 No 50 0 4 3 3 2 5 4 5 3 3 3 2 5 42 19 3 3 7 0 32 74 84 562510
540880 50 84 540880
59 563226 No 50 0 4 2 2 1 2 2 1 1 1 1 2 3 22 0 3 3 6 0 12 34 44 563226
580147 50 44 580147
60 567391 No 50 0 3 0 1 0 1 1 0 0 1 1 0 1 9 0 3 3 6 0 12 21 31 567391
559933 50 31 559933
61 568467 No 100 0 2 1 1 1 0 0 0 0 0 0 1 3 9 0 0 0 0 0 0 9 29 568467
62 574968 No 50 0 3 1 2 0 2 1 1 0 2 1 0 2 15 0 4 3 4 0 11 26 36 574968
526049 50 36 526049
63 580540 No 50 0 2 1 1 0 3 2 1 0 0 1 0 4 15 11 3 2 6 0 22 37 47 580540
554678 50 47 554678
64 587348 No 50 0 3 1 1 0 2 2 1 1 1 0 1 1 14 0 3 3 6 0 12 26 36 587348
522702 50 36 522702
65 587931 No 50 0 3 1 1 0 2 2 1 0 1 1 1 3 16 7 3 3 6 0 19 35 45 587931
513102 50 45 513102
66 594991 No 50 0 4 2 2 1 3 2 1 2 3 2 1 4 27 20 3 4 6 2 35 62 72 594991
520976 50 72 520976
67 595576 DONE 50 10 3 1 1 1 2 2 1 1 1 1 1 2 17 0 3 3 6 0 12 19 29 595576
329943 50 29 329943
68 624385 DONE 100 0 4 2 2 1 3 3 2 1 2 3 3 3 29 13 3 3 6 2 27 56 76 624385
69 637899 No 60 0 4 2 2 1 2 2 1 1 1 1 1 3 21 19 4 3 6 0 32 53 65 637899
660025 40 61 660025
70 649218 No 50 0 3 2 3 1 4 3 1 1 2 3 2 3 28 20 4 3 6 0 33 61 71 649218
544851 50 71 544851
71 650451 No 50 0 4 3 2 1 2 2 1 1 2 2 2 3 25 0 3 4 7 0 14 39 49 650451
635411 50 49 635411
72 668579 DONE 100 0 1 0 1 0 0 0 0 0 0 0 0 0 2 0 2 3 5 -5 5 7 27 668579
73 677839 No 33 0 3 2 2 1 2 3 1 1 2 2 3 1 23 0 3 2 5 0 10 33 33 677839
537948 33 33 537948
542586 33 33 542586
GRIDVIVACONTRIBPEN-
ALTY
1234567 89101112 SUM 13 14151617SUMGR
MARK
TOTALID
MARK FOR REPORTMARK FOR PROGRAM


Some comments on marking the Introduction to AI assignments

Report

The report was marked according to the main criteria: Idea (10%), Description of the implementation (5%), Evaluation (15%) , and Assessment (25%). Another 5% were for the references. While most students did reasonably well on the first two categories, unfortunately the evaluation was often difficult to understand. Essentially we expected:

Code

Code was assessed using the Spam Assassin corpus, filtered to remove those emails using an encoding not supported by Java, giving about 5000 emails total. A 20% hold-out was used. Ideally we would have used 10- or 5-times cross-validation, but doing so would have taken too long (it took several days to run all the filters just with hold-out).

For the 20 marks for performance of code, 1/3 were given for the percentage of spam emails classified correctly, and the remaining for the percentage of genuine emails classified correctly. Submitting working code was the easiest way to get lots of marks in this assignment. A working naive Bayes filter would normally achieve at least 15 out of 20. Unfortunately many submissions did not work. The most common problem was an ArrayIndexOutOfBounds exception. Briefly, the cause for this is as follows:

We suggested in several emails that everyone read the code in the provided spam-filter.scm. In this file there are the lines:

  ;; Create an experiment
  (define experiment
    ;;(cross-validate 10)) ;; 10-fold cross validation
    ;;(hold-out 0.2))      ;; 20% holdout
    (same-test-and-train)) ;; test and train on same data

Testing on the same data used in training is clearly a bad idea. Had hold out or cross validation been used this problem would have been caught before submission.

The remaining 20 marks were given for code organisation and style. Most of the submissions were naive Bayes filters with minimal enhancements over the code developed in tutorials. It was disappointing to see such little originality and it was quite difficult to mark code where the only original contribution was a few lines of code. Novel implementations did receive a small bonus.


© Manfred Kerber, 2004, Introduction to AI
25.5.2005
The URL of this page is http://www.cs.bham.ac.uk/~mmk/Teaching/AI/local/ai-cont-assessment.html.
URL of module http://www.cs.bham.ac.uk/~mmk/Teaching/AI/