CHALLENGES IN DETERMINING TERM RELEVANCE FOR TEXT DATA

A. Rehman, K. Javed, H. A. Babri

Abstract


Although the nature of text data is different from ordinary non-text datasets in a number of ways, existing algorithms from Machine Learning domain have been borrowed for the classification of text data. Machine learning algorithms cannot be readily applied on raw text data. Text data needs to be transformed to a suitable form for the application of machine learning algorithms. The transformation produces further problems for feature selection and classification algorithms. In this paper we highlight the problems introduced by transformation of text data. We also show how different feature selection algorithms including bi-normal separation, information gain and ROC are affected by text data.

Full Text:

PDF

References


References

Li-Ping Jing, H.K. Huang and H.B. Shi, Proc. of

Int. Conf. on Improved Feature Selection

Approach to tfidf in Text Mining, Machine

Learning and Cybernetics 2 (2002) pp. 944-946.

P.M. Ciarelli, E. Oliveira, C. Badue and A.F. De

Souza, International Journal of Computer

Information Systems and Industrial Management

Applications 1 (2009) 133.

X.B. Xue and Z.H. Zhou, IEEE Transactions on

Knowledge and Data Engineering 21, No. 3

(2009) 428.

H. Moisl, Data Normalization for Variation in

Document Length in Exploratory Multivariate

Analysis of Text Corpora (2008).

A. Singhal, G. Salton and C. Buckley, Length

Normalization in Degraded Text Collections,

Proceedings of Fifth Annual Symposium on

Challenges in determining term relevance for text data 185

Document Analysis and Information Retrieval,

(1995) p. 1517.

T. Joachims. Proc. of 24th Annual Int. ACM

SIGIR Conference on Research and Development

in Information Retrieval, NY, USA (2001) p. 128.

M. Baroni, 39 distributions in Text, University of

Trento (2005). http://clic.cimec.unitn.it/marco/

publications/hsk_39_dist_rev2.pdf

M.J. Saary, Journal of Clinical Epidemiology 61,

No. 4 (2008) 311.

D. Fouarge and R. Muffers, Social Exclusion in

European Welfare States, urn:nbn:nl:ui:27-21326.

(2002).

David D. Lewis. Reuters-21578.

G. Forman, Proceedings of the 21st Int. Conf. on

Machine Learning, ACM (2004) p. 38.

G. Forman, I. Guyon and A. Elisseeff, Journal of

Machine Learning Research 3 (2003) 12891305.

Y. Yang and J.O. Pedersen, Proc. of the 14th Int.

Conf on Machine Learning, San Francisco, CA,

USA (1997) p. 412.


Refbacks

  • There are currently no refbacks.