Continuous Assessment Introduction to AI

DISCLAIMER:

All marks are provisional, may be changed upon further scrutiny, and need approval by various boards of examiners. There are given here exclusively in order to give a first indication and without any guarantee.

Since we expect higher achievements from three member teams than from two member teams and the group mark does not reflect this, we add 10 to the group mark to compute the individual marks in a two member team.

Marks are scaled with respect to the contribution of each team member in the range of plus minus 10 marks so that the average of the marks of the group members is the group mark. Special rules apply in the case of minor contributions of a group member.

You can find general comments on the marking at the end of this file.

Marks for Report only Assignments

GR	ID	VIVA	CONTRIB	PEN- ALTY	MARK FOR REPORT							GR MARK	TOTAL	ID
					1	2	3	4	5	6	7
1	323508	No	50	0	18	18	9	8	10	9	3	75	85	323508
	592346		50										85	592346
2	329172	No	50	0	12	10	8	7	12	7	1	57	67	329172
	326786		50										67	326786
3	625335	No	100	0	10	8	12	5	9	5	2	51	71	625335

Marks for Report plus Implementation Assignments

GR	ID	VIVA	CONTRIB	PEN- ALTY	MARK FOR REPORT												SUM	MARK FOR PROGRAM					SUM	GR MARK	TOTAL	ID
					1	2	3	4	5	6	7	8	9	10	11	12		13	14	15	16	17
4	322903	No	33	0	2	1	1	0	2	3	2	3	3	3	2	3	25	0	4	4	7	0	15	40	40	322903
	543376		33																						40	543376
	545198		33																						40	545198
5	323104	No	50	0	4	1	2	1	4	3	4	3	4	5	3	2	36	14	2	4	6	0	26	62	72	323104
	330383		50																						72	330383
6	328274	DONE	33	5	5	2	2	1	4	4	3	2	2	3	1	3	32	20	3	4	6	0	33	60	60	328274
	329460		33																						60	329460
	653612		33																						60	653612
7	328905	No	70	0	3	1	2	1	3	2	1	2	3	3	3	2	26	15	5	3	5	0	28	54	68	328905
	326099		30																						60	326099
8	329017	DONE	50	5	5	1	2	1	3	2	2	3	3	2	3	2	29	15	3	3	6	-4	23	47	57	329017
	329831		50																						57	329831
9	330319	No	37	5	4	2	1	1	3	4	2	3	3	4	3	2	32	0	5	3	8	0	16	43	45	330319
	328143		37																						45	328143
	325824		26																						39	325824
10	330366	No	50	0	6	3	2	1	3	3	2	3	5	4	1	0	33	14	2	2	5	0	23	56	66	330366
	326515		50																						66	326515
11	330612	No	50	0	3	1	1	1	1	1	1	1	2	2	2	3	19	0	2	2	4	0	8	27	37	330612
	328338		50																						37	328338
12	330684	No	50	5	3	1	2	0	2	3	1	2	4	2	1	2	23	7	3	3	6	0	19	37	47	330684
	587540		50																						47	587540
13	514190	DONE	50	0	2	1	1	1	1	0	0	0	1	1	2	2	12	0	0	2	4	2	8	20	30	514190
	540475		50																						30	540475
14	515297	No	50	0	4	2	2	1	4	3	4	3	4	4	3	3	37	20	2	3	5	0	30	67	77	515297
	530442		50																						77	530442
15	517206	No	50	0	3	1	2	1	3	3	2	2	3	3	4	3	30	18	3	3	5	0	29	59	69	517206
	532440		50																						69	532440
16	517571	No	33	0	3	1	2	1	4	3	3	3	3	3	2	2	30	0	3	3	6	0	12	42	42	517571
	524982		33																						42	524982
	558328		33																						42	558328
17	521178	No	50	0	3	2	1	1	2	3	1	2	4	3	4	2	28	18	3	3	5	0	29	57	67	521178
	543728		50																						67	543728
18	522586	DONE	50	5	3	1	2	1	2	2	1	2	2	3	2	2	23	0	2	4	6	0	12	30	40	522586
	537037		50																						40	537037
19	523904	No	50	0	6	3	3	2	5	4	5	3	6	7	4	3	51	18	2	1	3	0	24	75	85	523904
	540640		50																						85	540640
20	524732	No	50	0	7	2	2	1	3	3	1	3	6	6	4	2	40	0	3	3	6	3	15	55	65	524732
	525148		50																						65	525148
21	525308	No	50	0	6	2	3	2	4	4	2	3	4	5	3	3	41	18	3	3	6	0	30	71	81	525308
	527219		50																						81	527219
22	525502	No	50	0	2	1	1	1	0	0	0	0	0	0	2	1	8	7	0	0	0	0	7	15	25	525502
	518833		50																						25	518833
23	525574	No	50	0	3	2	2	1	0	1	0	0	2	2	2	0	15	0	2	3	5	0	10	25	35	525574
	595371		50																						35	595371
24	525716	DONE	50	0	4	2	2	2	4	3	4	3	4	5	3	3	39	20	2	3	6	-10	21	60	70	525716
	326044		50																						70	326044
25	527539	No	60	0	3	0	1	0	0	0	0	0	0	0	1	3	8	0	3	3	6	0	12	20	32	527539
	540516		40																						28	540516
26	528972	No	50	0	6	2	2	2	3	3	0	1	2	1	0	4	26	16	4	4	8	0	32	58	68	528972
	580571		50																						68	580571
27	529053	No	40	0	2	1	3	1	2	3	1	2	2	3	0	1	21	0	1	3	3	0	7	28	32	529053
	548495		30																						26	548495
	542054		30																						26	542054
28	529821	No	70	5	6	3	2	1	3	3	1	2	3	4	1	2	31	0	4	4	7	0	15	41	55	529821
	536000		30																						47	536000
29	530241	No	50	0	3	2	1	1	2	2	1	2	1	1	1	3	20	0	1	2	3	0	6	26	36	530241
	514483		50																						36	514483
30	530428	DONE	40	0	6	2	2	1	2	1	0	1	1	1	1	4	22	16	4	4	7	0	31	53	57	530428
	572582		40																						57	572582
	522296		20																						45	522296
31	531507	No	50	0	2	1	1	1	1	1	0	1	1	1	0	0	10	14	2	3	5	0	24	34	44	531507
	533076		50																						44	533076
32	532540	No	50	0	6	3	2	2	4	4	2	2	5	4	4	3	41	18	3	3	6	0	30	71	81	532540
	523982		50																						81	523982
33	533293	No	33	0	2	1	1	1	2	2	2	2	2	2	1	4	22	18	4	3	4	0	29	51	51	533293
	559930		33																						51	559930
	324334		33																						51	324334
34	533706	No	33	0	4	2	1	1	2	3	1	1	2	3	2	2	24	14	3	4	8	0	29	53	53	533706
	538649		33																						53	538649
	515175		33																						53	515175
35	536510	No	50	0	3	2	2	1	2	2	1	1	3	3	1	2	23	0	3	3	7	0	13	36	46	536510
	541494		50																						46	541494
36	536645	No	100	5	4	2	2	1	3	3	1	2	3	3	0	2	26	0	4	4	6	0	14	35	55	536645
37	536690	No	100	0	3	1	2	1	2	1	1	1	2	2	0	2	18	18	4	4	8	0	34	52	72	536690
38	536702	No	100	0	3	2	1	1	2	3	1	1	2	1	3	4	24	0	3	3	7	0	13	37	57	536702
39	537907	No	54	0	6	3	2	2	4	4	2	3	5	6	4	4	45	18	4	4	6	0	32	77	88	537907
	530995		46																						86	530995
40	538023	DONE	33	0	4	2	2	2	3	3	1	2	1	3	3	2	28	19	4	4	7	-5	29	57	57	538023
	557586		33																						57	557586
	549800		33																						57	549800
41	538432	DONE	50	0	4	2	2	2	2	3	1	1	2	2	3	4	28	0	4	4	7	2	17	45	55	538432
	536483		50																						55	536483
42	539808	No	50	0	3	2	1	1	2	2	1	1	3	3	2	3	24	0	3	4	7	0	14	38	48	539808
	529279		50																						48	529279
43	540631	No	50	0	6	2	2	2	3	3	1	2	2	3	3	3	32	0	3	3	6	0	12	44	54	540631
	534033		50																						54	534033
44	541537	No	50	0	3	1	2	1	4	4	2	3	4	5	4	3	36	16	3	4	7	0	30	66	76	541537
	511770		50																						76	511770
45	542571	No	55	0	5	3	2	1	2	3	1	1	3	2	2	0	25	0	3	3	6	0	12	37	48	542571
	517100		45																						46	517100
46	543718	No	33	0	3	1	2	1	2	2	1	1	2	1	2	1	19	0	3	3	6	0	12	31	31	543718
	329659		33																						31	329659
	548496		33																						31	548496
47	544501	No	50	0	6	3	2	1	4	4	2	2	4	4	3	3	38	0	3	4	7	0	14	52	62	544501
	542431		50																						62	542431
48	544833	DONE	41	0	3	2	3	2	4	4	1	2	5	4	4	5	39	0	2	3	6	0	11	50	55	544833
	543375		30																						48	543375
	328335		29																						48	328335
49	549993	No	50	0	3	2	2	1	2	1	1	1	1	2	2	4	22	0	3	3	6	0	12	34	44	549993
	530044		50																						44	530044
50	550737	DONE	50	10	4	2	2	1	2	2	1	1	1	2	1	3	22	0	3	3	6	0	12	24	34	550737
	551450		50																						34	551450
51	552536	No	50	0	6	3	3	2	4	4	1	2	3	5	2	3	38	19	3	3	6	2	33	71	81	552536
	537123		50																						81	537123
52	552974	No	90	0	6	2	2	2	4	4	1	3	3	4	2	2	35	18	3	3	4	0	28	63	81	552974
	329392		10																						15	329392
53	555361	No	66	0	6	2	2	2	5	5	1	2	5	5	3	3	41	0	3	3	6	0	12	53	66	555361
	662476		33																						60	662476
54	556708	No	100	0	4	1	2	1	3	3	1	2	2	3	2	3	27	0	3	3	6	0	12	39	59	556708
55	558350	No	50	0	4	3	2	2	4	4	1	2	4	3	1	3	33	16	3	3	6	-5	23	56	66	558350
	643441		50																						66	643441
56	558502	No	34	0	4	2	2	1	4	3	4	2	3	2	1	2	30	15	3	3	6	0	27	57	58	558502
	544839		33																						57	544839
	580762		33																						57	580762
57	560175	DONE	50	0	3	2	3	2	4	4	5	2	3	2	2	3	35	20	3	3	6	-10	22	57	67	560175
	536755		50																						67	536755
58	562510	No	50	0	4	3	3	2	5	4	5	3	3	3	2	5	42	19	3	3	7	0	32	74	84	562510
	540880		50																						84	540880
59	563226	No	50	0	4	2	2	1	2	2	1	1	1	1	2	3	22	0	3	3	6	0	12	34	44	563226
	580147		50																						44	580147
60	567391	No	50	0	3	0	1	0	1	1	0	0	1	1	0	1	9	0	3	3	6	0	12	21	31	567391
	559933		50																						31	559933
61	568467	No	100	0	2	1	1	1	0	0	0	0	0	0	1	3	9	0	0	0	0	0	0	9	29	568467
62	574968	No	50	0	3	1	2	0	2	1	1	0	2	1	0	2	15	0	4	3	4	0	11	26	36	574968
	526049		50																						36	526049
63	580540	No	50	0	2	1	1	0	3	2	1	0	0	1	0	4	15	11	3	2	6	0	22	37	47	580540
	554678		50																						47	554678
64	587348	No	50	0	3	1	1	0	2	2	1	1	1	0	1	1	14	0	3	3	6	0	12	26	36	587348
	522702		50																						36	522702
65	587931	No	50	0	3	1	1	0	2	2	1	0	1	1	1	3	16	7	3	3	6	0	19	35	45	587931
	513102		50																						45	513102
66	594991	No	50	0	4	2	2	1	3	2	1	2	3	2	1	4	27	20	3	4	6	2	35	62	72	594991
	520976		50																						72	520976
67	595576	DONE	50	10	3	1	1	1	2	2	1	1	1	1	1	2	17	0	3	3	6	0	12	19	29	595576
	329943		50																						29	329943
68	624385	DONE	100	0	4	2	2	1	3	3	2	1	2	3	3	3	29	13	3	3	6	2	27	56	76	624385
69	637899	No	60	0	4	2	2	1	2	2	1	1	1	1	1	3	21	19	4	3	6	0	32	53	65	637899
	660025		40																						61	660025
70	649218	No	50	0	3	2	3	1	4	3	1	1	2	3	2	3	28	20	4	3	6	0	33	61	71	649218
	544851		50																						71	544851
71	650451	No	50	0	4	3	2	1	2	2	1	1	2	2	2	3	25	0	3	4	7	0	14	39	49	650451
	635411		50																						49	635411
72	668579	DONE	100	0	1	0	1	0	0	0	0	0	0	0	0	0	2	0	2	3	5	-5	5	7	27	668579
73	677839	No	33	0	3	2	2	1	2	3	1	1	2	2	3	1	23	0	3	2	5	0	10	33	33	677839
	537948		33																						33	537948
	542586		33																						33	542586
GR	ID	VIVA	CONTRIB	PEN- ALTY	1	2	3	4	5	6	7	8	9	10	11	12	SUM	13	14	15	16	17	SUM	GR MARK	TOTAL	ID
					MARK FOR REPORT													MARK FOR PROGRAM

Some comments on marking the Introduction to AI assignments

Report

The report was marked according to the main criteria: Idea (10%), Description of the implementation (5%), Evaluation (15%) , and Assessment (25%). Another 5% were for the references. While most students did reasonably well on the first two categories, unfortunately the evaluation was often difficult to understand. Essentially we expected:

• A clear description of the ideas implemented, focusing on the own contribution. While it is good practice to put things in a context, it isn't to spend pages on standard material which everybody knows anyway (assumed they attended the lectures).
• The description of the implementation should have been a high level description, which allows anybody to implement the ideas themselves and arrive at the same results WITHOUT having to look at your code.
• In the evaluation we expected a description which shows how good your implementation classifies spam. Just to give a number such as 88% isn't good, since it does not make clear how this number has been achieved and how it can be reproduced. In other words, your description should have been so clear that everybody who uses your implementation can rerun your experiments and should get a result which is close to yours. Unfortunately many of you were pretty vague as to how they obtained the values. It is also an experimental weakness if you test your method on the same examples on which you trained them. You should have used either a holdout, or better cross-validation.
• The assessment bit was assumed to be the most difficult part. How robust are the results? Why are the results as they are? How can they be improved? Except from the last question most did not go in any of these matters.

Code

Code was assessed using the Spam Assassin corpus, filtered to remove those emails using an encoding not supported by Java, giving about 5000 emails total. A 20% hold-out was used. Ideally we would have used 10- or 5-times cross-validation, but doing so would have taken too long (it took several days to run all the filters just with hold-out).

For the 20 marks for performance of code, 1/3 were given for the percentage of spam emails classified correctly, and the remaining for the percentage of genuine emails classified correctly. Submitting working code was the easiest way to get lots of marks in this assignment. A working naive Bayes filter would normally achieve at least 15 out of 20. Unfortunately many submissions did not work. The most common problem was an ArrayIndexOutOfBounds exception. Briefly, the cause for this is as follows:

• With a 20% holdout the filter is trained on 80% of the corpus and tested on the remaining 20%.
• When the emails are bagged each unique word is assigned a corresponding unique integer that indexes into the occurrence and conditional probability arrays.
• With a hold-out some words will occur in the test set that did not occur in the training set. Hence they will be assigned integers greater than all seen during testing. Attempting to use these integers as indexes into the various arrays will cause an ArrayIndexOutOfBounds exception.

We suggested in several emails that everyone read the code in the provided spam-filter.scm. In this file there are the lines:

  ;; Create an experiment
  (define experiment
    ;;(cross-validate 10)) ;; 10-fold cross validation
    ;;(hold-out 0.2))      ;; 20% holdout
    (same-test-and-train)) ;; test and train on same data

Testing on the same data used in training is clearly a bad idea. Had hold out or cross validation been used this problem would have been caught before submission.

The remaining 20 marks were given for code organisation and style. Most of the submissions were naive Bayes filters with minimal enhancements over the code developed in tutorials. It was disappointing to see such little originality and it was quite difficult to mark code where the only original contribution was a few lines of code. Novel implementations did receive a small bonus.