ToxicDocs
0 likes111 views
The document discusses a repository of 4 million newly declassified legal documents from Columbia and CPR that are of interest to various groups. The author aims to make the documents more accessible by categorizing them, enabling retrieval of similar documents, inferring missing attributes, and visualizing trends over time. An example is shown of searching for and visualizing trends in documents containing "vinyl" over time. The author also discusses the classification and visualization techniques used, including using SVMs and feedforward neural networks.
ToxicDocs
- 2. ToxicDocs Background âą Columbia and CPR have 4 million newly declassified legal documents âą Of interest to journalists, historians, attorneys, public health officials âą How can this repository be made more accessible and useful? âą More generally, what can we do to get insights out of a document dump?
- 3. ToxicDocs I aim to: 1. Categorize documents into types such as memo, ad, scientific study 2. Provide retrieval of documents similar to a given one 3. Infer missing attributes (such as the year) by parsing text 4. Visualize trends for topics over time
- 4. The Actual Data DIVISION OF SAFETY ANIj . n -IENE The Industrial Commission of Ohio 700 W. Third Avenue, Columbus, Ohio 43212 .e~2e70 ~ .e~.sslle IlIdullri,1 COOt",lIlio. M. IIOlLAWO nlSE ChI"m,. mm l. tHOOU Go'ftrttt ""i,,,.DIYII;'" of wllty ",41 101m t. wl.mEt MtIlMr lHOI1W0,,,',I1f.11G,,.,U,,,U",GHU ~ ~ ~L.yI0Il" P. SHEOIAN
- 5. OCR Quality âą Out-of-Vocabulary (OOV) text, i.e. meaningless gibberish, is limited in most documents âą Most documents have >60% readable text
- 6. Classification Task âą 50k documents âą 1k randomly labeled âą 15 unbalanced classes âą Documents can be: âą subtle âą handwritten âą illustrations âą not in English âą blank (?)
- 7. Classification Algorithm Model âą Linear kernel one-v-all SVM (liblinear) âą Squared hinge loss + L2 regularization Performance âą Accuracy: 62% (top 3: 88%), mean F1: 0.57 âą Baseline: 15% using regular expressions Features âą Sublinear TF-IDF on n-gram features (56%) âą +6% from document length, NER, and semi-supervised label propagation
- 8. Visual Features âą Visual structure contains independent information âą 1-layer feed-forward network gets an accuracy of 15% âą Recent advances allow OCR with probabilistic models for language using recurrent networks 1000 x 1000 100 x 100 10 x 10
- 9. Putting it to Use âą A search for âvinylâ (H2C=CHCl) reveals the following trend in the composition of documents over time âą Time (x) is inferred and type (y) is predicted
- 10. Evolution of a Crisis âą chemicalindustryarchives.org/dirtysecrets/ âą âBy 1971 the industry knew without doubt that vinyl chloride caused cancer in animals.â
- 11. Evolution of a Crisis âą âIn January 1974, B.F. Goodrich announced the presence of a rare liver cancer, angiosarcoma, in its polyvinyl chloride workers at is Louisville plant.â
- 12. Evolution of a Crisis âą âIn May of 1974, the Occupational Safety and Health Administration (OSHA) proposed a maximum exposure level for vinyl chloride at a no detectable levelâ
- 13. toxicdocs.org âą Launching some time after the election âą This work will be scaled and integrated âą github.com/GautamShine/toxic-docs About me: âą Ph.D. candidate at Stanford in electrical engineering âą Interned at an AI startup (ML/NLP) and twice at Intel (supercomputing) âą Outdoor enthusiast (climbing, diving) and backpacker (30+ countries) ToxicDocs
- 14. Semi-supervised Learning âą General idea: make use of unlabeled data âą For SVMs, distance from separating hyperplane serves as prediction confidence âą Procedure used: 1. h1 = argmax f(Xtrain) 2. Ć·unlabeled = h1(Xunlabeled) 3. Xabsorbed = {x â Xunlabeled | h1(x) > C} 4. h2 = argmax f(Xtrain + Xabsorbed) 5. Ć·test = h2(Xtest)
- 15. Kernels for Text âą n-gram space is high-d and sparse âą Most points lie in a low-d subspace âą Data is often trivially separable âą Projection to higher dimensions doesnât reduce bias much âą But pays a price with higher variance âą Big speed difference in NLP problems âą Linear kernel is O(p) from SGD on hinge loss âą Non-linear kernels are O(p2) from coordinate ascent on the Lagrange dual