This page links to the latest versions of course material. Some PDF, some HTML. Update May 29, 2018
Lecture Notes (chronological order)
- BDA18-D3 Chap9_CART RB. For the class of April 9, on CART.
- BDA18 Class 4 Lecture notes Toyota For the class of April 11, on CART + Toyota
- Logistic Regression 2018 Class of April 16 on classification using linear models aka logistic regression.
- Class of April 18 on linear categorical models aka logistic regression. BDA18 illustration of Rattle use 04-18
- Notes on Linear Regression, Week 4, April 23, 25 BDA18 regression 04/25.pdf. BDA18 regression slides 4-23. Use primarily the April 25 version; 4/23 has a few additional slides.
- How to go from Rattle to R. BDA18 Rattle to R code 4-25.pdf
- Lecture Notes Week 5 Random Forests BDA18 Random Forests2018B
- Lecture Notes Week 6 Text Mining, Day 1
Tutorial worked through in class. Basic Text Mining in R 2017 version
- Week 6 Text mining #2 2018b
- Week 7 LASSO, Monday May 14.
- Week 8 lecture notes. Monday May 21. BDA18 feature engineering case study
Advice, tutorials, reference books, other useful material
Special topics – for specific papers
The Big Data Analytics course introduces data mining with techniques and concepts that are broadly applicable. Individual topics and projects have specific techniques, needs, and resources. In keeping with the theme “Borrow and re-use, don’t invent anything yourself,” here are some resources that are especially suited to particular topics.
Don’t forget to try to site’s Search window (usually near the upper right) to look up possible keywords. Many of these topics also have entire books about them, such as on Springerlink.
- Especially useful R books for the course. Resources for Mining + R language
- Text processing. Start with this list: Text Mining Resources for Projects Then look at https://bda2020.files.wordpress.com/2017/04/bda17-text-mining-resources.pdf These two pages alone will save many hours of programming time. There are also many books on this subject. Specific books include: Mining Text Data R for Marketing Research and Analytics
- Spatial data, Geographic Information Systems. For projects on taxis, bicycle sharing, crime, and many other topics where the underlying data is geographically distributed, and location affects behavior. Read this page: Spatial (GIS) data in R: easy maps One of many books is Applied Spatial Data Analysis with R. Also Spatial analysis in R
- Time series require a special kind of validation, in which you train the model on early years, and then validate it on later years. You can do this in rolling fashion. For example use years 1-5 for training, and validate on year 6. Then use years 1 to 6 for training (or 2 to 6), and then validate on year 7. Validating machine learning time series models
- Twitter and other social networking sites. In addition to material on text mining, R for Marketing Research and Analytics; Text mining of Amazon reviews.; Also be sure to read about “Regular Expressions.” Handling and Processing Strings in R by Gason Sanchez is a 100 page mini- book on manipulating text. Look here when you need to do something with text like “find all words that start with ‘UCSD’.” Finally, there are many previous student papers in BDA that use Twitter data.
- Local crime. Local crime models are tricky because they require predicting events that are spread out over space and time. If you set up your data with “buckets” that are geographically and temporally small, then most buckets are empty. But if you make the buckets too large, such as “Any time on Mondays, for the lower half of Manhattan,” then the buckets are too big to be useful to decision makers. Wk 8: Feature engineering, other topics CHRONological handouts, 2016. Lectures 2017
Google folder for the course. There you will find all datasets for the textbook,
The official textbook web site is http://www.dataminingbook.com/book/r-edition
Once you register, you can get these datasets, and the R Code. (It’s better to type the R Code by hand, the first time.)
PROFESSOR ROGER BOHN OFFICE = RBC 1315 PHONE 858 534-7630
EMAIL: RBOHNat UCSDdotEDU.
Personal web site: Art2science.org
Here are the lecture notes on Random Forests from Thursday May 11. BDA17 Random Forests May 11 Bohn Remember, Random Forests are a technique everyone should try. LASSO, also discussed on Wednesday, is great when you have lots of variables. With fewer than 20 variables, it’s not as necessary. BUT
LASSO, also discussed on Wednesday, is great when you have lots of variables. With fewer than 20 variables, it’s not as necessary. BUT remember that you will often want to add interaction terms (and jump terms/quadratic terms/etc.) to linear models. As soon as you start that, the number of variables ballons.