?>

EMNLP 2014: Conference on Empirical Methods in Natural Language Processing — October 25–29, 2014 — Doha, Qatar.

emnlp2014

To register for CodeSwitch Shared Task, click below:

REGISTER

First Workshop on Computational Approaches to Code Switching

Tweet Level


Codeswitched tweets :
2665 |
Monolingual tweets :
209 |
Total tweets :
2874
Team Accuracy Recall Precision F1-Score
JustAnEagerStudent 0.951 0.980 0.968 0.974
MSR-India 0.953 0.989 0.962 0.975
IUCL 0.912 0.956 0.949 0.952
dcu-uvt 0.958 0.994 0.961 0.977
CMU 0.930 0.945 0.979 0.962
A3-107 0.948 0.979 0.966 0.972
Baseline (LangID) 0.421 0.395 0.951 0.559
Baseline (Lexical) 0.904 0.949 0.947 0.948

Codeswitched tweets :
429 |
Monolingual tweets :
1197 |
Total tweets :
1626
Team Accuracy Recall Precision F1-Score
CMU0.8950.7670.8230.794
Baseline (LangID)0.7680.3870.5930.468
Baseline (Lexical)0.7080.8600.4710.608
dcu-uvt0.8380.9440.6290.755
TAU0.9070.8160.8290.823
IUCL0.8710.5780.8990.704
JustAnEagerStudent0.7130.9460.4780.635
A3-1070.8730.8720.7110.783
MSR-India0.8530.8530.6750.754

Codeswitched tweets :
247 |
Monolingual tweets :
66 |
Total tweets :
313
Team Accuracy Recall Precision F1-Score
CMU 0.815 0.931 0.849 0.888
IUCL 0.824 0.943 0.850 0.894
A3-107 0.751 0.814 0.863 0.838
MSR-India 0.818 0.955 0.837 0.892
Baseline (Lexical) 0.607 0.615 0.844 0.712

Codeswitched tweets :
32 |
Monolingual tweets :
2300 |
Total tweets :
2332
Team Accuracy Recall Precision F1-Score
MSR-India 0.947 0.344 0.097 0.152
IUCL 0.974 0.125 0.111 0.118
A3-107 0.605 0.719 0.025 0.048
CMU 0.861 0.531 0.052 0.095
Baseline (Lexical) 0.757 0.531 0.030 0.057

Codeswitched tweets :
293 |
Monolingual tweets :
1484 |
Total tweets :
1777
Team Accuracy Recall Precision F1-Score
CMU0.6620.7340.2920.417
MSR-INDIA0.7140.2120.1830.196
Baseline (Lexical)0.5130.7750.2210.344
A3-1070.4690.8230.2130.338
IUCL0.7660.2490.2710.260
GWU*0.4400.9560.2220.360
CMU*0.6150.7320.3470.471

Token Level

Nepali English


lang1 :
12286 |
lang2 :
17216 |
mixed :
60 |
ne :
1071 |
other :
9635 |
Total :
40268
Team Recall Precision F1-Score
JustAnEagerStudent 0.944 0.949 0.947
MSR-India 0.967 0.929 0.948
IUCL 0.851 0.891 0.871
dcu-uvt 0.979 0.952 0.965
CMU 0.916 0.948 0.932
A3-107 0.954 0.932 0.943
Baseline (LangID) 0.571 0.765 0.654
Baseline (Lexical) 0.997 0.507 0.672

lang1 :
12286 |
lang2 :
17216 |
mixed :
60 |
ne :
1071 |
other :
9635 |
Total :
40268
Team Recall Precision F1-Score
JustAnEagerStudent 0.965 0.964 0.965
MSR-India 0.980 0.959 0.969
IUCL 0.689 0.976 0.808
dcu-uvt 0.988 0.961 0.974
CMU 0.869 0.964 0.914
A3-107 0.984 0.955 0.969
Baseline (LangID) 0.923 0.628 0.747
Baseline (Lexical) 0.640 0.936 0.760

lang1 :
12286 |
lang2 :
17216 |
mixed :
60 |
ne :
1071 |
other :
9635 |
Total :
40268
Team Recall Precision F1-Score
JustAnEagerStudent 0.000 1.000 0.000
MSR-India 0.000 0.000 0.000
IUCL 0.017 1.000 0.033
dcu-uvt 0.033 0.500 0.063
CMU 0.500 0.012 0.023
A3-107 0.000 1.000 0.000
Baseline (LangID) 0.000 1.000 0.000
Baseline (Lexical) 0.000 1.000 0.000

lang1 :
12286 |
lang2 :
17216 |
mixed :
60 |
ne :
1071 |
other :
9635 |
Total :
40268
Team Recall Precision F1-Score
JustAnEagerStudent 0.510 0.657 0.574
MSR-India 0.350 0.644 0.454
IUCL 0.551 0.487 0.517
dcu-uvt 0.456 0.804 0.582
CMU 0.421 0.599 0.494
A3-107 0.390 0.791 0.522
Baseline (LangID) 0.000 1.000 0.000
Baseline (Lexical) 0.000 1.000 0.000

lang1 :
12286 |
lang2 :
17216 |
mixed :
60 |
ne :
1071 |
other :
9635 |
Total :
40268
Team Recall Precision F1-Score
JustAnEagerStudent 0.968 0.935 0.951
MSR-India 0.955 0.990 0.972
IUCL 0.961 0.208 0.342
dcu-uvt 0.958 0.991 0.974
CMU 0.953 0.959 0.956
A3-107 0.935 0.957 0.946
Baseline (LangID) 0.549 0.915 0.686
Baseline (Lexical) 0.448 0.996 0.618

lang1 :
12286 |
lang2 :
17216 |
mixed :
60 |
ne :
1071 |
other :
9635 |
Total :
40268
Team Overall Accuracy
JustAnEagerStudent 0.946
MSR-India 0.952
IUCL 0.752
dcu-uvt 0.963
CMU 0.890
A3-107 0.946
Baseline (LangID) 0.700
Baseline (Lexical) 0.685

Spanish English


ambiguous :
41 |
lang1 :
7424 |
lang2 :
5278 |
mixed :
13 |
ne :
374 |
other :
4289 |
Total :
17419
Team Recall Precision F1-Score
CMU0.0240.0480.032
Baseline (LangID)0.0001.0000.000
Baseline (Lexical)0.0001.0000.000
dcu-uvt0.0240.0770.037
TAU0.0240.2000.043
IUCL0.0000.0000.000
JustAnEagerStudent0.0000.0000.000
A3-1070.0001.0000.000
MSR-India0.0001.0000.000

ambiguous :
41 |
lang1 :
7424 |
lang2 :
5278 |
mixed :
13 |
ne :
374 |
other :
4289 |
Total :
17419
Team Recall Precision F1-Score
CMU0.9420.9230.933
Baseline (LangID)0.8910.7470.812
Baseline (Lexical)0.9910.6210.764
dcu-uvt0.9590.9150.936
TAU0.9660.9390.952
IUCL0.9520.9300.941
JustAnEagerStudent0.9240.8560.889
A3-1070.9410.9170.929
MSR-India0.9610.9240.942

ambiguous :
41 |
lang1 :
7424 |
lang2 :
5278 |
mixed :
13 |
ne :
374 |
other :
4289 |
Total :
17419
Team Recall Precision F1-Score
CMU0.9200.9510.936
Baseline (LangID)0.8530.7570.802
Baseline (Lexical)0.4240.8980.576
dcu-uvt0.9250.9290.927
TAU0.9520.9520.952
IUCL0.9300.9350.932
JustAnEagerStudent0.8170.8930.853
A3-1070.9060.9350.920
MSR-India0.9440.9320.938

ambiguous :
41 |
lang1 :
7424 |
lang2 :
5278 |
mixed :
13 |
ne :
374 |
other :
4289 |
Total :
17419
Team Recall Precision F1-Score
CMU0.0000.0000.000
Baseline (LangID)0.0001.0000.000
Baseline (Lexical)0.0001.0000.000
dcu-uvt0.0001.0000.000
TAU0.0001.0000.000
IUCL0.0001.0000.000
JustAnEagerStudent0.0001.0000.000
A3-1070.0001.0000.000
MSR-India0.0001.0000.000

ambiguous :
41 |
lang1 :
7424 |
lang2 :
5278 |
mixed :
13 |
ne :
374 |
other :
4289 |
Total :
17419
Team Recall Precision F1-Score
CMU0.2410.4810.321
Baseline (LangID)0.0001.0000.000
Baseline (Lexical)0.0001.0000.000
dcu-uvt0.3640.6900.476
TAU0.4810.6720.561
IUCL0.4870.6250.547
JustAnEagerStudent0.2380.6850.353
A3-1070.2700.5400.360
MSR-India0.1950.5250.285

ambiguous :
41 |
lang1 :
7424 |
lang2 :
5278 |
mixed :
13 |
ne :
374 |
other :
4289 |
Total :
17419
Team Recall Precision F1-Score
CMU0.9390.8900.914
Baseline (LangID)0.4900.8030.609
Baseline (Lexical)0.7110.9860.827
dcu-uvt0.9260.9530.939
TAU0.9420.9540.948
IUCL0.9240.9290.927
JustAnEagerStudent0.9420.9110.926
A3-1070.9240.8800.902
MSR-India0.9430.9600.952

ambiguous :
41 |
lang1 :
7424 |
lang2 :
5278 |
mixed :
13 |
ne :
374 |
other :
4289 |
Total :
17419
Team Overall Accuracy
CMU0.917
Baseline (LangID)0.759
Baseline (Lexical)0.726
dcu-uvt0.925
TAU0.942
IUCL0.926
JustAnEagerStudent0.879
A3-1070.909
MSR-India0.932

Mandarin English


lang1 :
4703 |
lang2 :
881 |
mixed :
1 |
ne :
254 |
other :
442 |
Total :
6281
Team Recall Precision F1-Score
CMU 0.980 0.979 0.980
IUCL 0.983 0.978 0.981
A3-107 0.977 0.979 0.978
MSR-India 0.984 0.976 0.980
Baseline (Lexical) 0.990 0.824 0.900

lang1 :
4703 |
lang2 :
881 |
mixed :
1 |
ne :
254 |
other :
442 |
Total :
6281
Team Recall Precision F1-Score
CMU 0.832 0.662 0.737
IUCL 0.839 0.666 0.742
A3-107 0.674 0.663 0.669
MSR-India 0.891 0.666 0.762
Baseline (Lexical) 0.381 0.613 0.470

lang1 :
4703 |
lang2 :
881 |
mixed :
1 |
ne :
254 |
other :
442 |
Total :
6281
Team Recall Precision F1-Score
CMU 0.000 0.000 0.000
IUCL 0.000 1.000 0.000
A3-107 0.000 1.000 0.000
MSR-India 0.000 1.000 0.000
Baseline (Lexical) 0.000 1.000 0.000

lang1 :
4703 |
lang2 :
881 |
mixed :
1 |
ne :
254 |
other :
442 |
Total :
6281
Team Recall Precision F1-Score
CMU 0.740 0.542 0.626
IUCL 0.701 0.503 0.586
A3-107 0.839 0.384 0.527
MSR-India 0.677 0.652 0.664
Baseline (Lexical) 0.000 1.000 0.000

lang1 :
4703 |
lang2 :
881 |
mixed :
1 |
ne :
254 |
other :
442 |
Total :
6281
Team Recall Precision F1-Score
CMU 0.215 0.812 0.340
IUCL 0.183 0.920 0.306
A3-107 0.215 0.714 0.330
MSR-India 0.210 0.949 0.344
Baseline (Lexical) 0.176 0.975 0.299

lang1 :
4703 |
lang2 :
881 |
mixed :
1 |
ne :
254 |
other :
442 |
Total :
6281
Team Overall Accuracy
MSR-India 0.904
IUCL 0.895
CMU 0.896
A3-107 0.875
Baseline (Lexical) 0.808

Modern Arabic Dialets


ambiguous :
11 |
lang1 :
44134 |
lang2 :
141 |
ne :
5939 |
other :
3902 |
Total :
54127
Team Recall Precision F1-Score
MSR-India 0.000 0.000 0.000
IUCL 0.000 0.000 0.000
A3-107 0.000 0.000 0.000
CMU 0.000 0.000 0.000
Baseline (Lexical) 0.000 1.000 0.000

ambiguous :
11 |
lang1 :
44134 |
lang2 :
141 |
ne :
5939 |
other :
3902 |
Total :
54127
Team Recall Precision F1-Score
MSR-India 0.965 0.921 0.942
IUCL 0.961 0.816 0.882
A3-107 0.924 0.953 0.938
CMU 0.922 0.970 0.946
Baseline (Lexical) 0.987 0.868 0.924

ambiguous :
11 |
lang1 :
44134 |
lang2 :
141 |
ne :
5939 |
other :
3902 |
Total :
54127
Team Recall Precision F1-Score
MSR-India 0.532 0.093 0.158
IUCL 0.348 0.089 0.142
A3-107 0.397 0.031 0.057
CMU 0.574 0.049 0.090
Baseline (Lexical) 0.206 0.041 0.068

ambiguous :
11 |
lang1 :
44134 |
lang2 :
141 |
ne :
5939 |
other :
3902 |
Total :
54127
Team Recall Precision F1-Score
MSR-India 0.470 0.748 0.577
IUCL 0.033 0.234 0.058
A3-107 0.702 0.770 0.734
CMU 0.778 0.706 0.740
Baseline (Lexical) 0.000 1.000 0.000

ambiguous :
11 |
lang1 :
44134 |
lang2 :
141 |
ne :
5939 |
other :
3902 |
Total :
54127
Team Recall Precision F1-Score
MSR-India 0.841 0.994 0.911
IUCL 0.004 0.019 0.006
A3-107 0.897 0.853 0.874
CMU 0.988 0.973 0.981
Baseline (Lexical) 0.821 0.992 0.898

ambiguous :
11 |
lang1 :
44134 |
lang2 :
141 |
ne :
5939 |
other :
3902 |
Total :
54127
Team Overall Accuracy
MSR-India 0.901
IUCL 0.788
CMU 0.910
A3-107 0.896
Baseline (Lexical) 0.864

Modern Arabic Dialets Test 2


ambiguous :
119 |
lang1 :
10459 |
lang2 :
14800 |
mixed :
2 |
ne :
4321 |
other :
2940 |
Total :
32641
Team Recall Precision F1-Score
CMU0.0000.0000.000
MSR-INDIA0.0080.0560.015
Baseline (Lexical)0.0001.0000.000
A3-1070.0000.0000.000
IUCL0.0000.0000.000
GWU*0.0001.0000.000
CMU*0.0000.0000.000

ambiguous :
119 |
lang1 :
10459 |
lang2 :
14800 |
mixed :
2 |
ne :
4321 |
other :
2940 |
Total :
32641
Team Recall Precision F1-Score
CMU0.8540.6900.763
MSR-INDIA0.9620.4220.587
Baseline (Lexical)0.9870.3790.547
A3-1070.9130.4700.620
IUCL0.9070.4380.590
GWU*0.8990.6200.734
CMU*0.8440.6110.709

ambiguous :
119 |
lang1 :
10459 |
lang2 :
14800 |
mixed :
2 |
ne :
4321 |
other :
2940 |
Total :
32641
Team Recall Precision F1-Score
CMU0.7610.8730.813
MSR-INDIA0.3600.8430.505
Baseline (Lexical)0.1630.8920.276
A3-1070.3440.8790.494
IUCL0.4770.7830.593
GWU*0.6230.8880.732
CMU*0.7410.8940.811

ambiguous :
119 |
lang1 :
10459 |
lang2 :
14800 |
mixed :
2 |
ne :
4321 |
other :
2940 |
Total :
32641
Team Recall Precision F1-Score
CMU0.0001.0000.000
MSR-INDIA0.0001.0000.000
Baseline (Lexical)0.0001.0000.000
A3-1070.0001.0000.000
IUCL0.0000.0000.000
GWU*0.5000.0050.010
CMU*0.0001.0000.000

ambiguous :
119 |
lang1 :
10459 |
lang2 :
14800 |
mixed :
2 |
ne :
4321 |
other :
2940 |
Total :
32641
Team Recall Precision F1-Score
CMU0.6870.7880.734
MSR-INDIA0.2830.8420.424
Baseline (Lexical)0.0001.0000.000
A3-1070.5850.7970.675
IUCL0.0850.2860.131
GWU*0.8710.9710.918
CMU*0.6550.7280.689

ambiguous :
119 |
lang1 :
10459 |
lang2 :
14800 |
mixed :
2 |
ne :
4321 |
other :
2940 |
Total :
32641
Team Recall Precision F1-Score
CMU0.9820.9850.984
MSR-INDIA0.2950.8510.438
Baseline (Lexical)0.8980.9880.941
A3-1070.8010.7050.750
IUCL0.0110.0470.017
GWU*0.9930.9680.981
CMU*0.9860.9870.986

ambiguous :
119 |
lang1 :
10459 |
lang2 :
14800 |
mixed :
2 |
ne :
4321 |
other :
2940 |
Total :
32641
Team Overall Accuracy
CMU0.798
MSR-INDIA0.536
Baseline (Lexical)0.471
A3-1070.598
IUCL0.519
GWU*0.775
CMU*0.777