AI-First Engineering Cybertraining

• 1: Project Guidelines

This course introduces the students to AI-First Engineering Cybertraining we provide the following sections

Class Material

As part of this class, we will be using a variety of sources. To simplify the presentation we provide them in a variety of smaller packaged material including books, lecture notes, slides, presentations and code.

Note: We will regularly update the course material, so please always download the newest version. Some browsers try to be fancy and cache previous page visits. So please make sure to refresh the page.

We will use the following material:

Course Lectures and Management

Course Lectures. These meeting notes are updated weekly (Web)

Overview

This course is built around the revolution driven by AI and in particular deep learning that is transforming all activities: industry, research, and lifestyle. It will a similar structure to The Big Data Class and the details of the course will be adapted to the interests of participating students. It can include significant deep learning programming.

All activities – Industry, Research, and Lifestyle – are being transformed by Artificial Intelligence AI and Big Data. AI is currently dominated by deep learning implemented on a global pervasive computing environment - the global AI supercomputer. This course studies the technologies and applications of this transformation.

We review Core Technologies driving these transformations: Digital transformation moving to AI Transformation, Big Data, Cloud Computing, software and data engineering, Edge Computing and Internet of Things, The Network and Telecommunications, Apache Big Data Stack, Logistics and company infrastructure, Augmented and Virtual reality, Deep Learning.

There are new “Industries” over the last 25 years: The Internet, Remote collaboration and Social Media, Search, Cybersecurity, Smart homes and cities, Robotics. However, our focus is Traditional “Industries” Transformed: Computing, Transportation: ride-hailing, drones, electric self-driving autos/trucks, road management, travel, construction Industry, Space, Retail stores and e-commerce, Manufacturing: smart machines, digital twins, Agriculture and Food, Hospitality and Living spaces: buying homes, hotels, “room hailing”, Banking and Financial Technology: Insurance, mortgage, payments, stock market, bitcoin, Health: from DL for pathology to personalized genomics to remote surgery, Surveillance and Monitoring: – Civilian Disaster response; Miltary Command and Control, Energy: Solar wind oil, Science; more data better analyzed; DL as the new applied mathematics, Sports: including Sabermetrics, Entertainment, Gaming including eSports, News, advertising, information creation and dissemination, education, fake news and Politics, Jobs.

We select material from above to match student interests.

Students can take the course in either software-based or report-based mode. The lectures with be offered in video form with a weekly discussion class. Python and Tensorflow will be main software used.

Lectures on Particular Topics

Introduction to AI-Driven Digital Transformation

Introduction to AI-Driven Digital Transformation (Web)

Introduction to Google Colab

A Gentle Introduction to Google Colab (Web)

A Gentle Introduction to Python on Google Colab (Web)

MNIST Classification on Google Colab (Web)

MNIST-MLP Classification on Google Colab (Web)

MNIST-RNN Classification on Google Colab (Web)

MNIST-LSTM Classification on Google Colab (Web)

MNIST-Autoencoder Classification on Google Colab (Web)

MNIST with MLP+LSTM Classification on Google Colab (Web)

Distributed Training with MNIST Classification on Google Colab (Web)

PyTorch with MNIST Classification on Google Colab (Web)

Material

Health and Medicine

Health and Medicine sector has become a much more needed service than ever. With the uprising of the Covid-19, resource usage, monitoring, research on anti-virals and many more challenging tasks were on the shoulders of scientists. To face such challenges, AI can become a worthy partner in solving some of the related problems efficiently and effectively.

AI in Banking

AI in banking has become a vital component in providing best services to the peopel. AI provides securing bank transactions, providing suggestions and many other services for the clients. And legacy banking systems are also being reinforced with novel AI techniques to migrate business models with technology.

Space and Energy

Energy is a term we find in everyday life. Conserving energy and smart usage is vital in managing energy demands. Here the role played by AI has become significant in recent years. Many efforts have been taken by industry leaders like Bill Gates to provide better solutions for efficient energy consumption. Apart from that Space explorations are also being reinforced with AI. Better communication, remote sensing, data analysis have become key components in succeeding the challenge to unravel the mysteries in the universe.

Mobility (Industry)

Mobility is a key part in everyday life. From the personal car to space exploring rockets, there are many places that can be enhanced by using AI. Autonomous vehicles and sensing features provide safety and efficiency. Many motorcar companies have already moved towards AI to power the vehicles and provide new features for the drivers.

Cloud Computing

Cloud computing is a major component of Today's service infrastructures. Artificial intelligence, micro-services, storage, virtualization and parallel computing are some of the key aspects of cloud computing.

Commerce

Commerce is a field which is reinforced with AI and technologies to provide a better service to the clients. Amazon is one of the leading companies in e-commerce. The recommendation engines play a major role in e-commerce.

Complementary Material

• When working with books, ePubs typically display better than PDF. For ePub, we recommend using iBooks on macOS and calibre on all other systems.

Piazza

Piazza. The link for all those that participate in the IU class to its class Piazza.

Scientific Writing with Markdown

Scientific Writing with Markdown (ePub) (PDF)

Git Pull Request

Git Pull Request. Here you will learn how to do a simple git pull request either via the GitHub GUI or the git command line tools

Introduction to Linux

This course does not require you to do much Linux. However, if you do need it, we recommend the following as starting point listed

The most elementary Linux features can be learned in 12 hours. This includes bash, editor, directory structure, managing files. Under Windows, we recommend using gitbash, a terminal with all the commands built-in that you would need for elementary work.

Introduction to Linux (ePub) (PDF)

Older Course Material

Older versions of the material are available at

Lecture Notes 2020 (ePub) (PDF)

Big Data Applications (Nov. 2019) (ePub) (PDF)

Big Data Applications (2018) (ePub) (PDF)

Contributions

You can contribute to the material with useful links and sections that you find. Just make sure that you do not plagiarize when making contributions. Please review our guide on plagiarism.

Computer Needs

This course does not require a sophisticated computer. Most of the things can be done remotely. Even a Raspberry Pi with 4 or 8GB could be used as a terminal to log into remote computers. This will cost you between $50 - $100 dependent on which version and equipment. However, we will not teach you how to use or set up a Pi or another computer in this class. This is for you to do and find out.

In case you need to buy a new computer for school, make sure the computer is upgradable to 16GB of main memory. We do no longer recommend using HDD’s but use SSDs. Buy the fast ones, as not every SSD is the same. Samsung is offering some under the EVO Pro branding. Get as much memory as you can effort. Also, make sure you back up your work regularly. Either in online storage such as Google, or an external drive.

1 - Project Guidelines

All students of this class are doing a software project. (Some of our classes allow non software projects)

• The final project is 50% of grade

• All projects must have a well-written report as well as the software component

• We must be able to run software from class GitHub repository. To do so you must include an appendix to your project report describing how to run your project.

  • If you use containers you must decsribe how to create them from Docker files.
  • If you usue ipy notebooks you must include a button or links so it can be run in Google collab

• There are a useful set of example projects submitted in previous classes

  • Last year’s Fall 2020 Big Data class
  • Earlier Classes [https://github.com/cybertraining-dsc/pub/blob/master/docs/vonLaszewski-cloud-vol-9.pdf] (26 projects)
  • Earlier Classes [https://github.com/cybertraining-dsc/pub/blob/master/docs//vonLaszewski-i523-v3.pdf] (45 projects)
  • Earlier Classes [https://github.com/cybertraining-dsc/pub/blob/master/docs//vonLaszewski-i524-spring-2017.pdf] (28 projects)

• In this class you do not have the option to work on a joint report, however you can collaborate.

• Note: all reports and projects are open for everyone as they are open source.

Details

The major deliverable of the course is a software project with a report. The project must include a programming part to get a full grade. It is expected that you identify a suitable analysis task and data set for the project and that you learn how to apply this analysis as well as to motivate it. It is part of the learning outcome that you determine this instead of us giving you a topic. This topic will be presented by student in class April 1.

It is desired that the project has a novel feature in it. A project that you simply reproduce may not recieve the best grade, but this depends on what the analysis is and how you report it.

However “major advances” and solving of a full-size problem are not required. You can simplify both network and dataset to be able to complete project. The project write-up should describe the “full-size” realistic problem with software exemplifying an instructive example.

One goal of the class is to use open source technology wherever possible. As a beneficial side product of this, we are able to distribute all previous reports that use such technologies. This means you can cite your own work, for example, in your resume. For big data, we have more than 1000 data sets we point to.

Comments on Example Projects from previous classes

Warning: Please note that we do not make any quality assumptions to the published papers that we list here. It is up to you to identify outstanding papers.

Warning: Also note that these activities took place in previous classes, and the content of this class has since been updated or the focus has shifted. Especially chapters on Google Colab, AI, DL have been added to the course after the date of most projects. Also, some of the documents include an additional assignment called Technology review. These are not the same as the Project report or review we refer to here. These are just assignments done in 2-3 weeks. So please do not use them to identify a comparison with your own work. The activities we ask from you are substantially more involved than the technology reviews.

Format of Project

Plagiarism is of course not permitted. It is your responsibility to know what plagiarism is. We provide a detailed description book about it here, you can also do the IU plagiarism test to learn more.

All project reports must be provided in github.com as a markdown file. All images must be in an images directory. You must use proper citations. Images copied from the Internet must have a citation in the Image caption. Please use the IEEE citation format and do not use APA or harvard style. Simply use fotnotes in markdown but treat them as regular citations and not text footnotes (e.g. adhere to the IEEE rules).
All projects and reports must be checked into the Github repository. Please take a look at the example we created for you.

The report will be stored in the github.com.

./project/index.md

./project/images/mysampleimage.png

Length of Project Report

Software Project Reports: 2500 - 3000 Words.

Possible sources of datasets

Given next are links to collections of datasets that may be of use for homework assignments or projects.

• [https://www.data.gov/] [https://github.com/caesar0301/awesome-public-datasets]

• [https://aws.amazon.com/public-data-sets/]

• [https://www.kaggle.com/datasets]

• [https://cloud.google.com/bigquery/public-data/github]

• [https://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public]

• [http://pascal.inrialpes.fr/data/human/]

• [https://github.com/fivethirtyeight/uber-tlc-foil-response]

• [http://www.census.gov/population/www/cen2010/glance/]

• [https://ieee-dataport.org/datasets]

• [https://worlddata.ai/]

FAQ

• Why you should not just paste and copy into the GitHub GUI?

  We may make comments directly in your markdown or program files. If you just paste and copy you may overlook such comments. HEns only paste and copy small paragraphs. If you need to. The best way of using github is from commandline and using editors such as pycharm and emacs.

• I like to do a project that relates to my company?

  • Please go ahead and do so but make sure you use open-source data, and all results can be shared with everyone. If that is not the case, please pick a different project.

• Can I use Word or Google doc, or LaTeX to hand in the final document?

  • No. you must use github.com and markdown.

  • Please note that exporting documents from word or google docs can result in a markdown file that needs substantial cleanup.

• Where do I find more information about markdown and plagiarism

• https://laszewski.github.io/publication/las-20-book-markdown/

• [https://cloudmesh-community.github.io/pub/vonLaszewski-writing.pdf]{.ul}

• Can I use an online markdown editor?

  • There are many online markdown editors available. One of them is [https://dillinger.io/]{.ul}.
    Use them to write your document or check the one you have developed in another editor such as word or google docs.

  • Remember, online editors can be dangerous in case you lose network connection. So we recommend to develop small portions and copy them into a locally managed document that you then check into github.com.

  • Github GUI (recommended): this works very well, but the markdown is slightly limited. We use hugo’s markdown.

  • pyCharm (recommended): works very well.

  • emacs (recommended): works very well

• What level of expertise and effort do I need to write markdown?

  • We taught 10-year-old students to use markdown in less than 5 minutes.

• What level of expertise is needed to learn BibTeX

  • We have taught BibTeX to inexperienced students while using jabref in less than an hour (but it is not required for this course). You can use footnotes while making sure that the footnotes follow the IEEE format.

• How can I get IEEE formatted footnotes?

  • Simply use jabref and paste and copy the text it produces.

• Will there be more FAQ’s?

  • Please see our book on markdown.

  • Discuss your issue in piazza; if it is an issue that is not yet covered, we will add it to the book.

• How do I write URLs?

  • Answered in book

  • Note: All URL’s must be either in [TEXT](URLHERE) or <URLHERE> format.