Project Guidelines

All students of this class are doing a software project. (Some of our classes allow non software projects)

• The final project is 50% of grade

• All projects must have a well-written report as well as the software component

• We must be able to run software from class GitHub repository. To do so you must include an appendix to your project report describing how to run your project.

  • If you use containers you must decsribe how to create them from Docker files.
  • If you usue ipy notebooks you must include a button or links so it can be run in Google collab

• There are a useful set of example projects submitted in previous classes

  • Last year’s Fall 2020 Big Data class
  • Earlier Classes [https://github.com/cybertraining-dsc/pub/blob/master/docs/vonLaszewski-cloud-vol-9.pdf] (26 projects)
  • Earlier Classes [https://github.com/cybertraining-dsc/pub/blob/master/docs//vonLaszewski-i523-v3.pdf] (45 projects)
  • Earlier Classes [https://github.com/cybertraining-dsc/pub/blob/master/docs//vonLaszewski-i524-spring-2017.pdf] (28 projects)

• In this class you do not have the option to work on a joint report, however you can collaborate.

• Note: all reports and projects are open for everyone as they are open source.

Details

The major deliverable of the course is a software project with a report. The project must include a programming part to get a full grade. It is expected that you identify a suitable analysis task and data set for the project and that you learn how to apply this analysis as well as to motivate it. It is part of the learning outcome that you determine this instead of us giving you a topic. This topic will be presented by student in class April 1.

It is desired that the project has a novel feature in it. A project that you simply reproduce may not recieve the best grade, but this depends on what the analysis is and how you report it.

However “major advances” and solving of a full-size problem are not required. You can simplify both network and dataset to be able to complete project. The project write-up should describe the “full-size” realistic problem with software exemplifying an instructive example.

One goal of the class is to use open source technology wherever possible. As a beneficial side product of this, we are able to distribute all previous reports that use such technologies. This means you can cite your own work, for example, in your resume. For big data, we have more than 1000 data sets we point to.

Comments on Example Projects from previous classes

Warning: Please note that we do not make any quality assumptions to the published papers that we list here. It is up to you to identify outstanding papers.

Warning: Also note that these activities took place in previous classes, and the content of this class has since been updated or the focus has shifted. Especially chapters on Google Colab, AI, DL have been added to the course after the date of most projects. Also, some of the documents include an additional assignment called Technology review. These are not the same as the Project report or review we refer to here. These are just assignments done in 2-3 weeks. So please do not use them to identify a comparison with your own work. The activities we ask from you are substantially more involved than the technology reviews.

Format of Project

Plagiarism is of course not permitted. It is your responsibility to know what plagiarism is. We provide a detailed description book about it here, you can also do the IU plagiarism test to learn more.

All project reports must be provided in github.com as a markdown file. All images must be in an images directory. You must use proper citations. Images copied from the Internet must have a citation in the Image caption. Please use the IEEE citation format and do not use APA or harvard style. Simply use fotnotes in markdown but treat them as regular citations and not text footnotes (e.g. adhere to the IEEE rules).
All projects and reports must be checked into the Github repository. Please take a look at the example we created for you.

The report will be stored in the github.com.

./project/index.md

./project/images/mysampleimage.png

Length of Project Report

Software Project Reports: 2500 - 3000 Words.

Possible sources of datasets

Given next are links to collections of datasets that may be of use for homework assignments or projects.

• [https://www.data.gov/] [https://github.com/caesar0301/awesome-public-datasets]

• [https://aws.amazon.com/public-data-sets/]

• [https://www.kaggle.com/datasets]

• [https://cloud.google.com/bigquery/public-data/github]

• [https://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public]

• [http://pascal.inrialpes.fr/data/human/]

• [https://github.com/fivethirtyeight/uber-tlc-foil-response]

• [http://www.census.gov/population/www/cen2010/glance/]

• [https://ieee-dataport.org/datasets]

• [https://worlddata.ai/]

FAQ

• Why you should not just paste and copy into the GitHub GUI?

  We may make comments directly in your markdown or program files. If you just paste and copy you may overlook such comments. HEns only paste and copy small paragraphs. If you need to. The best way of using github is from commandline and using editors such as pycharm and emacs.

• I like to do a project that relates to my company?

  • Please go ahead and do so but make sure you use open-source data, and all results can be shared with everyone. If that is not the case, please pick a different project.

• Can I use Word or Google doc, or LaTeX to hand in the final document?

  • No. you must use github.com and markdown.

  • Please note that exporting documents from word or google docs can result in a markdown file that needs substantial cleanup.

• Where do I find more information about markdown and plagiarism

• https://laszewski.github.io/publication/las-20-book-markdown/

• [https://cloudmesh-community.github.io/pub/vonLaszewski-writing.pdf]{.ul}

• Can I use an online markdown editor?

  • There are many online markdown editors available. One of them is [https://dillinger.io/]{.ul}.
    Use them to write your document or check the one you have developed in another editor such as word or google docs.

  • Remember, online editors can be dangerous in case you lose network connection. So we recommend to develop small portions and copy them into a locally managed document that you then check into github.com.

  • Github GUI (recommended): this works very well, but the markdown is slightly limited. We use hugo’s markdown.

  • pyCharm (recommended): works very well.

  • emacs (recommended): works very well

• What level of expertise and effort do I need to write markdown?

  • We taught 10-year-old students to use markdown in less than 5 minutes.

• What level of expertise is needed to learn BibTeX

  • We have taught BibTeX to inexperienced students while using jabref in less than an hour (but it is not required for this course). You can use footnotes while making sure that the footnotes follow the IEEE format.

• How can I get IEEE formatted footnotes?

  • Simply use jabref and paste and copy the text it produces.

• Will there be more FAQ’s?

  • Please see our book on markdown.

  • Discuss your issue in piazza; if it is an issue that is not yet covered, we will add it to the book.

• How do I write URLs?

  • Answered in book

  • Note: All URL’s must be either in [TEXT](URLHERE) or <URLHERE> format.