# Introduction Contributors: Alicia Wang, Conner Swenberg ### What is backend development? Backend Development is a subset of software development that specializes in the storage and transmission of data. A backend serves information to mobile, web, and other devices: whether you use Facebook on your phone or your computer, you communicate with the same place to get the information that is served on your screen. You can think of it as the “behind-the-scenes” of web functionality--it works together with the front end to deliver a product to the end user. Thus, backend development can encompass many things in a system, including (but not limited to) application logic, databases, and servers. ### Why you should take this course Backends sit at the core of an application. What makes Reddit Reddit is the vast network of related information that allow people to communicate over the internet. Banks and payment transfers like Venmo all have to be stored somewhere and validated by some protocol for security. Spotify and other streaming services need to organize and optimize content delivery to all devices to ensure a happy customer. The backbone of the internet and future innovations heavily rely on understanding and developing powerful backend systems. If these things excite your curiosity, you're in the right place! Taking the Intro to Backend Development course will give you the theoretical and experiential foundation to apply your creativity and build these systems yourself. ### What makes backend difficult Due to its “behind-the-scenes” nature, backend development can be a pain to debug. Unlike user-facing mobile apps or websites, there is less visual feedback to understand the problems within your code. There are also many possible ways to architect a system, but seeing the path towards the best way to design the application for functionality, simplicity, and maintainability is not always easy. # SP26 Syllabus Chimdi Ejiogu, Fanhao Yu ### Course Staff & Office Hours
NameRoleOffice HoursLocation
Claire YuTAMon 4:30 - 5:30pmHollister 320
Tran TranTAMon 4:30 - 5:30pmHollister 320
Sophie StrausbergTAMon 4:30 - 5:30pmHollister 320
Olivia YuTAMon 5:30 - 6:30pmHollister 320
Ashley HerreraTAMon 5:30 - 6:30pmHollister 320
Joshua DirgaTATue 4:30 - 5:30pmHollister 372
Lauren Ah-HotTATue 4:30 - 5:30pmHollister 372
Fanhao YuCourse InstructorTue 5:30 - 6:30pmHollister 372
Skye SlatteryTATue 5:30 - 6:30pmHollister 372
Andrew PungTAThu 4:30 - 5:30pmHollister 320
Chris VoonTAThu 4:30 - 5:30pmHollister 320
Chimdi EjioguCourse InstructorThu 4:30 - 5:30pmHollister 320
Anik DeyTAFri 3:00 - 4:00pmHollister 368
Yitbrek MataTAFri 3:00 - 4:00pmHollister 368
Wyatt CoxTAFri 3:00 - 4:00pmHollister 368
### Schedule {% hint style="info" %} This schedule is tentative and subject to changes to Cornell's Spring 2026 academic calendar. {% endhint %} Lectures are every **Monday** **and Wednesday** from **7:30 PM to 8:20 PM** in **Gates G01,** and each week's lectures will be split into two parts: a concept-based overview of the week's topic, followed by a live implementation demo of the same topic. Additionally, there will be a project assignment every week, which will typically be due on the **following** **Monday**. You can find the lecture recordings on our [YouTube channel](https://www.youtube.com/channel/UCUDPjOx1yPRGxsO6AFtBZSg) for you to rewatch as you wish. | | | | | | ------------------------------------- | ------------------------------------------------------ | ---------------------------------------------- | ----------------------------------- | | Lecture Dates | Topic | Project | Project Due Date | | 03/11, 03/16 | Routes | Reddit | 03/23 | | 03/21 | Git Workshop | N/A | - | | 03/18, 03/23 | Databases | Venmo (Basic) | 04/06 | | 03/25, 04/06 | Relational Databases | Venmo (Full) | 04/13 | | 04/08, 04/13 | Abstractions | CMS | 04/20 | | 04/15, 04/20 | Containerization | Dockerize CMS | 04/27 | | 04/20 | Hack Challenge Kickoff | Hack Challenge | N/A | | 04/22, 04/27 | Deployment | Deploy CMS | 05/04 | | 04/29 | Authentication + Images | N/A | N/A | | 05/04 | Developer Panel / Conclusion | N/A | N/A | | 05/04 | Hack Challenge Finale | N/A | N/A | ### Enrollment You can enroll yourself into the course like any other on Student Center upon passing the pre-test and submitting the application (see [Apply to Take the Course](/apply-to-take-the-course)). This is a 2-credit S/U course, however, we permit students to enroll for 1-credit for the purpose of avoiding a credit overload petition. All students are expected to do the same work, regardless of the number of enrolled credits. ### Weekly Assignments Programming assignments and weekly feedback forms are due on **Mondays** at **11:59 pm** and submitted on CMS. **Students will have 6 total slip days throughout the semester to submit assignments without a grading penalty. After all slip days have been used, there will be a -10% penalty per day on the grade of the given unsubmitted assignment. No submissions will be accepted after the late deadline for the assignment.** *Because of the slip day policy, no extensions will be granted.* Each assignment will have three optional challenges to take your understanding further. * Tier I challenges are designed to be low difficulty, simple extensions of the application (estimated time: <1 hour). * Tier II challenges are medium difficulty extensions that require more forethought for a successful implementation, but are by no means out of reach (estimated time: <2 hours). * Tier III challenges are high difficulty extensions which will often require outside research and will teach you things that we do not have time to cover in this course. We highly encourage you to reference demos, attend office hours, and ask questions on Ed discussion. Along with submitting assignment source code, you will be required to submit a text file (README.txt) where you can freely comment on the week’s lecture, assignment, or anything else. You will also fill out a short questionnaire which helps us quantify feedback for improving the course. ### Grading As mentioned earlier, this is a **2 credit S/U course,** and grading will be calculated as shown in the table below. Below is general information about the submission timeline for each assignment, although submission schedules for a given assignment may occasionally deviate from this timeline. Submission Deadline: **Monday, 11:59 pm**\ Late Deadline: **Wednesday, 11:59 pm**\ Late Penalty: **-1 point penalty per day. We will not accept homework after the late deadline.**\ Safety Net: **6 total slip days for the semester**, **up to 3 can be used per assignment.** | | | | ------------------- | ------- | | Item | Weight | | PA1 - Reddit | 10 | | PA2 - Venmo (Basic) | 10 | | PA3 - Venmo (Full) | 10 | | PA4 - CMS | 10 | | PA5 - Dockerize CMS | 10 | | PA6 - Deploy CMS | 10 | | Final Project | 30 | | Weekly Surveys | 5 | | Attendance | 5 | | **Total** | **100** | | **Passing Score** | **70** | ### Hack Challenge All AppDev courses finish with bringing members from iOS, backend, Android, and design together to put what you’ve learned to the test and build your very own application. All students are welcome to submit app ideas, and teams of up to four will be formed to bring the idea to life. Leading up to the Hack Challenge, the backend course will focus on deployment so that your mobile apps can connect to a live backend running on Google Cloud servers. It goes without saying that the majority of students find the Hack Challenge the most rewarding and fun part of the course! ### Academic Integrity As with any other course at Cornell, the code of academic integrity will be enforced in this class. While we encourage you to collaborate with other students, all code that you submit must be written by you (and although a great method for learning, pair programming does not abide by our standards of students submitting their own work). We also encourage using the internet to learn more about backend development, but again, any code you submit must be written by you. # Apply to Take the Course Contributors: Alanna Zhou ## Welcome! The course will officially start on March 11th, 2026. Applications will be open on February 2nd, and will close on March 8th, 11:59 PM EDT. Excited to teach you all for Spring 2026! Note: You can feel free to self-enroll in Student Center for 2 credits once you've received a submission comment that you're eligible to enroll :) ## 1. Submit Course Application A friendly reminder to apply if you haven't already! We use the application to evaluate how prepared our students are for the course, get some demographic data, and get the list of student NetIDs for adding people to CMS for submitting assignments. #### Course Application Here: ## 2. Submit assignment 0 (A0) In past semesters, we've had a very simple pretest in Python for students to submit and gauge their readiness for the course. In Spring 2020, we waived the pretest requirement and found that there were students who struggled with Python and would have benefited from taking the course a semester later or after they have had a bit more experience with coding. Unfortunately, codePost, where we once hosted the test, is no longer in service. Instead, we now have an identical completion-based assignment. Think of this as a helpful indicator of how ready you are to take the course (or what you need to do to be ready) and how you can balance your semester! {% hint style="info" %} If you have questions or any issues about this, don't hesitate to reach out to . {% endhint %} ### Download, unzip, and start coding in the pretest starter code You are **not allowed** to use the following functions: `sum()` , `min()`, and any kind of sorting function such as `list.sort()` as that defeats the purpose of the assignment! {% file src="/files/5EKV9DrYXsWgBZAorY7O" %} Assignment 0 Starter Code {% endfile %} {% hint style="warning" %} Don't be intimidated by the starter code! A0 only requires a ***very*** **basic** understanding of lists, classes, dictionaries, and loops, and you'll be surprised at how quickly you can complete it once you get started. Feel free to look things up online if you don't know something! If you are still concerned, don't worry; we've also had many past students successfully complete the course without ever having taken CS 1110. {% endhint %} ## 3. Enroll in Student Center Once you've completed A0, you can enroll on CS 1998 Sec 603 on Student Center anytime before March 11th at 11:59 PM. The course number is **16289**. {% hint style="danger" %} The add deadline of March 11th at 11:59 PM is a **very strict** deadline (7wk2 class add deadline), meaning that the registrar will not permit any exceptions. We have had issues in the past and were not able to grant everyone who took the course credit because of this, so please keep that in mind! {% endhint %} {% hint style="warning" %} The drop deadline is April 8th, 2026. We'll update this if anything changes. Refer to this [webpage](https://registrar.cornell.edu/calendar/2025-2026) for the key academic dates for this semester! If you are taking the course and are worried about passing, we'll give you an estimate before you need to drop! Almost all students pass, though, so this shouldn't be your worry at all :) {% endhint %} ## **4. Congrats and let's get ready!** {% content-ref url="/pages/-M16sflzQ7-eOmqr9r6j" %} [Getting Started](/getting-started) {% endcontent-ref %} # Getting Started Contributors: Conner Swenberg, Alanna Zhou ## 1. Read Course Information Pages {% content-ref url="/pages/-Lu9\_2GknJpEjD9Msryg" %} [Introduction](/) {% endcontent-ref %} {% content-ref url="/pages/kygfk9OXTyLTFmFPhuu3" %} [SP26 Syllabus](/sp26-syllabus) {% endcontent-ref %} ## 2. Apply to the course {% content-ref url="/pages/-MFaoTGXWzZTcnjt9gqj" %} [Apply to Take the Course](/apply-to-take-the-course) {% endcontent-ref %} ## 3. Enroll on Student Center Like all other courses, you must enroll yourself on Student Center before the add deadline. For your convenience, our course code is **16289**. This is not the permission number--this is just the code you use to find our course on Student Center. ## 4. Add Ed Discussion We use Ed to manage classroom communications and answering student questions. The course Ed discussion is linked on the left sidebar underneath `Getting Started.` Join the Ed here: {% hint style="danger" %} Even if you do not anticipate asking questions, Ed will be the main method of communication between students and course staff, so take advantage of it! {% endhint %} ## 5. Install & Get Acquainted with Recommended Tools Completing assignments will rely heavily on three tools: Python, a text editor, Postman, and the command line. ### Install Python Before you do anything, make sure that you have Python 3.9.12 installed. The reason is that our course runs testing scripts that use 3.9+ features. {% embed url="" %} Check your Python versions to make sure you'll be using the right one. Notice that if you have both Python2 and Python3, you may have to use `python3` when you're running it as a command so that your computer knows only to use version 3. ``` $ python --version Python 2.7.15 $ python -V Python 2.7.15 $ python3 --version Python 3.9.12 $ python3 -V Python 3.9.12 ``` ### Install a Text Editor {% embed url="" %} ### Install Postman {% content-ref url="/pages/-Lw\_IsDUKu2sa6KDel87" %} [Postman](/cheat-sheets/postman) {% endcontent-ref %} ### Know Command Line Basics {% content-ref url="/pages/-Lw\_J5gEPWGYjX\_aBA5n" %} [Command Line](/cheat-sheets/command-line) {% endcontent-ref %} ## 6. Prepare for Chapter 1 Our first lecture will introduce you to how computers communicate over the web and our demo will introduce you to the `Flask` python package which we will use throughout the course. We recommend watching a previous semester's lecture recording or reading the textbook chapter before coming to lecture. {% content-ref url="/pages/-Lu9\_81li-lENzx-dZaE" %} [1. Routes](/chapters/routes) {% endcontent-ref %} #### # Assignment Requirements Contributors: Shungo Najima ## Hi there! Here is a list of files/items you should submit with **every** programming assignment: 1. **Fill out your README.txt**\ A `README.txt` file is included in the starter code for you to fill out after completing the assignment. Also note that you will not receive extra credit for extra credit challenges you complete if you do not let us know for when we grade! 
Name: Jane Smith
   NetID: js123

   Challenges Attempted: <all the Tiers you completed here>
2. #### Make sure you have proper Python styling in your code Common mistakes in styling are: * NOT HAVING documentation/comments in each of your routes (a one-line comment is sufficient!) * Naming variables with CamelCase instead of snake\_case * Too much/too little empty spaces/empty lines * Leaving in commented code 3. #### **Verify your routes using** [**Postman Tests**](https://www.youtube.com/watch?v=V2ZWdPMBwSA) **and submit an exported JSON of them** 4. #### Zip and submit your assignment files Next, zip the starter folder and submit on CMS. For reference, your directory structure should look like the following: ``` pa1/ |-README.txt |-postman_collection.json |-src/ |-app.py |-requirements.txt ``` For clarification, this means that you **SHOULD NOT** include your virtual environment, pycache, or .db file in your final submission. Doing so will lose you a few points on the project. 5. **Submit Feedback Form** To receive credit for your assignment, you **must** fill out the [feedback form](https://forms.gle/9HhoJZjByiVxHdk76) so that we can better understand how effectively we are teaching and how long students are spending on assignments. # Assignment FAQs Contributors: Alanna Zhou ## Hi there! Here is a list of commonly asked questions on our past Ed Discussions. We've compiled this so that you can easily `ctrl+F` the error you are searching for quickly! Still feel free to pop that question on Piazza though, we're happy to help. If you have a specific error, you might benefit from this page: {% content-ref url="/pages/-MFNlUFN4w9BkJMHTErn" %} [Error FAQs 😢](/cheat-sheets/troubleshooting-errors) {% endcontent-ref %} ## PA1 * **Are comment ids unique within a post or globally?** * Globally! Regardless of which post a comment belongs to, no two comments should share the same id. * **Are we allowed to pre-populate our posts dictionary?** * Yes. Our grading is based around how well your application conforms to the provided API Specification. The initial state of your data structure has not impact on our tests. * **If a post has no comments, should we return an error or an empty list?** * Empty list. * **Why is the data in my GET response a different order from that of my dictionary? Does this matter?** * Python dictionaries are not ordered! This will not affect the correctness of your solution. * **If you can user query strings (/api/users/?user=username\&email=email), why use body requests?** * It's not sustainable to send large parameters, so more often than not we prefer POST. There could also also be security concerns if you could straight up see a password on the request URL ## **PA2** * **DB 'object' not callable?** * You probably have a `DB()` somewhere in your code where `DB = db.DatabaseDriver()` at the top of your file ## PA3 * **sqlite3.IntegrityError: FOREIGN KEY when deleting a user involved in a transaction?** * You can either delete all of the transactions that involve the user, or you can use something called a `SECONDARY KEY` (instead of `FOREIGN KEY`). We haven't taught you about cascade deletes (chapter 4 material!) so that's partly why you're getting the foreign key constraint failed error. You can think of a secondary key as a foreign key without a reference to a specific table (it's just an identifier for a row for any table in your database). This [**Stack Overflow post**](https://stackoverflow.com/questions/20668291/foreign-keys-vs-secondary-keys) might help. * **Does the user input the `accepted` field at `/api/transactions/`?** * The user does not have to give any value for `accepted`, or even put the `accepted` field into the POST body request. All this endpoint is doing is allowing the client to denote that there is a transaction that exists, but it is not necessarily a transaction that can go through. In other words, the transaction attempting to be created can be accepted later or denied later (if `accepted` is None, and the next endpoint in the API will allow the client to edit the transaction's `accepted` by `id`), or accepted now and denied now (`accepted` is `True` and then `False`). If you think about Venmo it makes sense; my friend can put in a request to pay me / request money from me, but the transaction will only go through if I accept the request to receive the payment or make the payment. The same goes the other way around if I make a request to my friend. ## PA4 * **What are backref and backpopulates and how do we use them?** {% content-ref url="/pages/-M4z72ioB6ognmybbUMi" %} [Concept FAQs](/cheat-sheets/concept-faqs) {% endcontent-ref %} * **How to categorize a User into instructor and student?** * Hint: Make two association tables, one for instructors/courses and one for students/courses. * **How is the assignment class is supposed to serialize its course without referencing the course class?** * You need to reference the course class. You can get the serialized course but omit any redundant data, for example, the assignment nested inside the course itself. ## PA5 * **Are we coding anything for our Docker image?** * Just create a Docker image with your PA4 source code! * **Do we submit on CMS and Docker Hub?** * Yes, and make sure that the `docker-compose.yml` file you submit on CMS is pointing to your image on Docker Hub. In other words, make sure that the image referenced inside of your `docker-compose.yml` matches the name of the Docker image that was built and pushed to DockerHub. * **How to check that `docker-compose.yml` file is pointing to Docker Hub image and not a local tag?** * You can delete the image off of your local machine with `docker image rm`. Then when you run `docker-compose up -d`, Docker will automatically pull the image off of Docker Hub! * **Can't pull Docker image from DockerHub?** * Check Docker Hub for your tag and see if it exists under your repository first. Your repository is just your username. Note that you should be trying to pull something that looks like `/`! # Error FAQs 😢 Contributors: Alanna Zhou * **No such file or directory: `requirements.txt`?** * Check that you are currently where `requirements.txt` exists. You'll know this is the case if enter `ls` (MacOS) or `dir` (Windows) into your terminal and you don't see `requirements.txt` or if you didn't spell it right in your command. * Or, you may be trying to do `pip3 install -r requirements.txt` inside of a correctly created and activated `venv`, but your `venv` is nested within a path that contains a folder that has a space in the name. Here's a good [**Stack Overflow post**](https://stackoverflow.com/questions/7911003/cant-install-via-pip-with-virtualenv) that explains the issue. * **No module named 'flask\_sqlalchemy'?** * Make sure you've run `pip3 install -r requirements.txt` in your activated virtual environment, check out the **Virtual Environment** page if you need more help! * **ImportError: cannot import name 'Feature' from 'setuptools'?** * Make sure that you have an updated `Markupsafe` version in your `requirements.txt` that you used to create virtual env: `MarkupSafe==1.1.1` * **How to fix "KeyError: 'data'" from running test suite?** * This means that your backend didn't respond with a format like:`{ "success": True, "data": blah }` so try making sure that your endpoints match the API spec linked in the backend course. * **Postman keeps giving me an error?** * Make sure you are hitting the correct URL in Postman. A correct URL looks like this: `http://localhost:5000/api/posts/{id}/comments/`, while some examples of incorrect ones look like this: `http://localhost:5000/api/posts/{id}/comments`, `http://localhost:5000/posts/{id}/comments/` * If you are making a POST request, make sure that you are submitting a JSON request. `Body` > `Raw` > `JSON` * **Postman 404 Not Found?** * Is your server running right now? Try checking your terminal. The server may stop when you make changes in your code has errors. * Also make sure that you have `if name == "main": app.run(host="0.0.0.0", port=5000, debug=True)` at the bottom of your `app.py`. * **TypeError: list indices must be integers or slices, not str?** * You may be returning a list, and accessing string values of that list (could be yourself or the testing script), instead of accessing string values of a dictionary. Make sure your responses abide by the API spec which indicates list/dictionary return types. * **JSONDecodeError?!** * `JSONDecodeError`s are among the most common in this course. It's caused when the testing script tries to parse the JSON data associated with a response (variable `res`) that does not have any JSON data associated with it. This occurs when your code in `app.py` encounters an uncaught exception and crashes before your crafted responses can be returned. To debug a `JSONDecodeError`, you need to look at which request is breaking on `res.json()` and look deeper into that route method in `app.py`. We then recommend using Postman to test that specific route. * Double check that you are using double quotes `"` when creating a POST body request. * Instead of accessing key values of a dictionary like `dict["key"]`, it's better to use `dict.get("key")` so that even if the POST request body doesn't contain the key, your code won't throw an error. * **Test file's `base_path = f"{LOCAL_URL}/api/user"` has `SyntaxError`?** * Our testing script uses what's called an f string. This is only supported by Python 3.6 or higher, so make sure you're running `python3 test.py`. * **Table X has no column named Y?** * Try using sqlite3 to help you debug in the terminal by 1\) opening sqlite3 shell: `sqlite3` 2\) loading your database file that you generate in your python code, which should show up in the same place as `app.py`: `.open .db` 3\) seeing all the tables in your database file `.tables` 4\) show what your table is made of (what the columns are): `PRAGMA table_info();` # Concept FAQs Contributors: Alanna Zhou ## backref vs. back\_populates vs. none of these This is introduced when we start using SQLAlchemy, and can be a bit confusing to understand why we need `backref` or `back_populates` at all and what the difference between these two are. To help you understand, I am going to break this up into three scenarios: 1. using neither `backref` nor `back_populates` 2. using `back_populates` 3. using `backref` Just as a note, we have been using `back_populates` in our demo and assignment solutions, but that does not mean you can't use `backref`! ### using neither `backref` nor `back_populates` Let's give an example of two classes that have a relationship with each other, `Parent` and `Child`. Let's assume that one parent can have multiple children, and each child can only have one parent. ```python class Parent(Base): __tablename__ = 'parent' id = Column(Integer, primary_key=True) children = relationship("Child") class Child(Base): __tablename__ = 'child' id = Column(Integer, primary_key=True) parent_id = Column(Integer, ForeignKey('parent.id')) parent = relationship("Parent") ``` This example will still "work", but is not using nor taking advantage of SQLAlchemy's recommended relationship pattern. You can read more about its one-to-many relationship pattern [here](https://docs.sqlalchemy.org/en/13/orm/basic_relationships.html#one-to-many). The consequence of this is that if you create child object A and assign it a parent object B, it does not mean that this parent object B will recognize child object A as its child: ```sql >>> parent = Parent() >>> child = Child() >>> child.parent = parent >>> print parent.children [] ``` As expected, this is wacky! `parent.children` should return me the newly created `child = Child()`! Let's fix this by using `back_populates`. ### using `back_populates` This is a SQLAlchemy keyword that you pass into the `relationship()` method that allows SQLAlchemy to recognize that there is a connection between the two entities that are related. To modify the previous example, we would do: ```python class Parent(Base): __tablename__ = 'parent' id = Column(Integer, primary_key=True) children = relationship("Child", back_populates="parent") class Child(Base): __tablename__ = 'child' id = Column(Integer, primary_key=True) parent_id = Column(Integer, ForeignKey('parent.id')) parent = relationship("Parent", back_populates="children") ``` The way that `back_populates` works is that you are directing each relationship from one end to the other: `Child` to `parent` (lowercase `p` in `parent`), and likewise `Parent` to `children`). Now, if you assign some children to a parent, the children will be able to recognize the parent. And if you assign a parent to a child, the parent will be able to recognize the child. By recognize, I am referring to having SQLAlchemy understand that the object is from an existing class you defined. The fruits of your labor can now be seen: ```python >>> parent = Parent() >>> child = Child() >>> child.parent = parent >>> print parent.children [Child(...)] ``` Solid! But what about `backref`? ### using `backref` This is simply another way of doing what `back_populates` does. There does not seem to be any execution advantage over the other (at least that Alanna knows of, please message her if you discover that that is not the case...). However, we recommend `back_populates` for a reason that is explained once you've read what exactly `back_ref` does. When using `backref`, you don't need to declare the relationship on the second table defined: ```python class Parent(Base): __tablename__ = 'parent' id = Column(Integer, primary_key=True) children = relationship("Child", backref="parent") class Child(Base): __tablename__ = 'child' id = Column(Integer, primary_key=True) parent_id = Column(Integer, ForeignKey('parent.id')) ``` Notice how `Child` did not have to use `relationship` nor `backref`; all it had to do was keep track of the `parent.id`. From seeing this, the difference between `backref` and `back_populates` is clear. Since you have to define the relationships in every class, we recommend using `back_populates` because it is nice and easy to see all the fields just be glancing at the model class, instead of having to look at other classes that define fields via `backref`. # Postman Contributors: Alanna Zhou ## Download Postman Highly recommend you install Postman as soon as the course starts! It will be crucial to assignments and practically any backend dev-ing. {% embed url="" %} ## Why use Postman If you hit an endpoint in your browser, the best you can do is hit endpoints that are just `GET` requests (see the section below on how to hit an endpoint). There's no easy way for you to test parts of your backend API that handle `POST` and `DELETE` requests -- so how do you know if you are responding in the way that you coded and expected? Use Postman! Postman lets you do all of these things with a super helpful user interface. You can even organize all of the endpoints your testing into folders and workspaces, as well as share them with whoever is collaborating with you on a project! ## How to "hit" an endpoint in general When you run your code locally (aka on local host), your endpoint at port `5000` can be reachable at all of these options: {% embed url="" %} {% embed url="" %} {% embed url="" %} This is because `0.0.0.0`, `127.0.0.1`, and `localhost` all mean you're running off of local host! You can read more about the differences on [this stack exchange post](https://superuser.com/questions/949428/whats-the-difference-between-127-0-0-1-and-0-0-0-0). #### Common mistake #1 If you can't hit your endpoint, ensure that you are also hitting the right port that your code is running on! This is usually declared somewhere at the bottom of your `app.py`: ```python if __name__ == '__main__': app.run(host='0.0.0.0', port=5000, debug=True) ``` #### Common mistake #2 There's a difference between `http` and `https`, so make sure you are hitting `http`! ## How to make a GET request in Postman 1. Select `GET` from the dropdown to the left of the `Enter request URL` input text field bar 2. Input the endpoint receiving the `GET` request into the text field bar 3. Click the blue `send` button 4. Postman will now show you either your successful `JSON` response at the bottom, or some errors for you to fix ### `GET` Query Parameters GET requests can also have parameters, and there are two ways to hit an endpoint with parameters: #### 1. Type in your `GET` parameters manually into the input text field bar {% embed url="" %} where `key1` and `key2` are the names of your parameters, and `value1` and `value2` are the corresponding values (you can have as many as you want, but this example only has two key-value pairs). For example, if I'm designing backend that returns some information about a student given their first name, last name, and age, I could type this URL into the text field input bar: {% embed url="" %} #### 2. Input your `GET` parameters into the table and let Postman automatically fill in your URL ![](/files/-Lweb9J5zXbWRsuFp3Bw) Just type away under `Params` (select this option in the banner under the URL) and the checkboxes will automatically be checked and your URL will automatically have the params filled in! ## How to make a POST request in Postman 1. Select `POST` from the dropdown to the left of the `Enter request URL` input text field bar 2. Input the endpoint receiving the `POST` request into the text field bar 3. Click on `Body` in the banner right below (next to `Headers` and `Pre-request Script`) 4. Select `raw` 5. Select `JSON` from the dropdown to the right of `raw` 6. Type in your `POST` request body in `JSON` format 7. Click the blue `send` button 8. Postman will now show you either your successful `JSON` response at the bottom, or some errors for you to fix Here's an example of what a `POST` request could look like, with the `{ "first" ... "true" }` information being the `POST` request body written in `JSON` format. ![](/files/-LweeXrhJjNri-_3PtlJ) Note that `JSON` can take in strings, integers, and booleans. # Command Line Contributors: Alanna Zhou ## The Directory Structure of an Operating System The **OS** (Operating System) on your computer organizes your files and folders in a hierarchy, all bundled up for each user. You may notice that you are the only user on your laptop, which is indicated by a folder that has the name of which you go by on your computer (in the diagram below, there happens to be four users) in some sort of `Users` folder. A **directory** is just a fancy word for folder, and **files** could mean anything with extensions like `.zip`, `.json`, .`txt`, `.mp3`, etc. ![](/files/-LxPRCZOfoD1cJfqtIsa) The command line comes in handy because it is a tool that allows you to jump in and out of directories (aka traversing this tree diagram), see what stuff is in them, and do cool things with just some commands you type! ## Why Use Command Line? #### It's Fast It can help you navigate your computer in much faster ways than clicking around on your mousepad on the graphical user interface. We highly recommend you get familiar with this throughout the course! #### It's Meant For Developing For example, if you want to be able to develop software in a virtual environment (which you can think about as some mode that you activate that allows you to get access to libraries that are relevant to the software you are currently developing), you do it through the command line! But don't worry about this yet, let's get through the basics first... ## The Basics of Command Line Although this tutorial is done with MacOS commands, it's quite similar to those of Windows and Linux. ### Traversing directories As mentioned earlier, the file system on your laptop is organized in a tree-like hierarchy, with files nested within folders within folders, all bundled up in a big folder that corresponds to a specific user. Let's assume that `user1` is the root directory. This means that if you are using `user1` on your laptop, when you open command line, you will automatically be in the directory `user1`. You can see what's in the directory that you are currently in with `ls` (this is the Windows equivalent to `dir`): ![](/files/-LxPZRmQz-HDVuD0CBhn) You can step further into directories (going down the tree in the diagram) with `cd ` (folder name must be one of those that were listed when you typed `ls` previously). Once you've entered `D1`, you can then type `ls` to see what's in `D1`: ![](/files/-LxPZkvv6FbPAg3NJsKZ) Oh, it's a text file named `f1`! However, I'm not ready to see this file yet. Let's try stepping back up the tree with `cd ..` (we just went from `user1` to `D1`, but let's go from `D1` to `user1`): ![](/files/-LxP_N0TzK8VVwqMggVz) And of course, when I type `ls`, we see that being back in `user1` properly gives us the folder `D1` that we saw when we first typed `ls`! And if I go further and further deeper down and go through more nested directories within `user1`, because we established earlier that `user1` is the root directory, I can always jump back to `user1` easily with simply: `cd`. ### Looking at file contents Now I'm ready to go back into `D1` and see what that `f1.txt` has with `cat `: ![](/files/-LxP_fHGoKgBF846xlrY) Great! Looks like `f1.txt` just contains a string `yeetaki mushroomz`. Now I want to see it with my default text editor instead of on the command line, and I can do that with `open `: ![](/files/-LxP_uaCceA6YS2uqKGs) And a new window should open with the file opened: ![](/files/-LxPa-Vc5MviU2cGJzUd) ## **MacOS & Linux Terminal** ### **The Basics** * look at what’s in the directory (aka folder) you are currently in: `ls` * see the path that leads you to the directory you are currently in: `pwd` * go into a directory: `cd ` * step back out of a directory: `cd ..` * jump to the root directory: `cd` ### Other helpful commands * make a directory: `mkdir ` * delete a directory: `rm -rf ` * delete a file: `rm ` * output contents of a file: `cat ` * to copy a file: `cp /` * open Finder from where you are currently: `open .` * open Visual Studio Code from where you are currently: `code .` ### Helpful tips to be faster * click on the up arrow to go through your history of commands * if you want to find a previous command and only know part of it (you don’t remember it completely), you can do: * `ctrl + r` and start typing to see the most recent command that matches what you’ve typed * to scroll through more suggestions, hold `ctrl` and hit `r` for each suggestion * to get your cursor to the beginning of what you just typed: `ctrl + a` * to get your cursor to the end of what you just typed: `ctrl + e` * remember it as `e` for “end” * delete line of text where your cursor is: `ctrl + u` * while you are in the middle of typing a folder name, or git branch name, you can press `tab` to autocomplete * if there are multiple suggestions, you can go through each suggestion by pressing `tab` each time ## Windows Command Prompt ### **The Basics** * look at what’s in the directory (aka folder) you are currently in: `dir` * see the path that leads you to the directory you are currently in: `pwd` * go into a directory: `cd ` * step back out of a directory: `cd ..` * jump to the root directory: `cd` ### Other helpful commands * make a directory: `mkdir ` * delete a directory: `rd /s /q ""` * delete a file: `rm ` * output contents of a file: `cat ` * to copy a file: `cp /` * open Windows Explorer from where you are currently: `start .` * open Visual Studio Code from where you are currently: `code .` # Virtual Environment Contributors: Alanna Zhou, Shungo Najima You can consult some [helpful Python docs](https://docs.python.org/3/tutorial/venv.html) for more information! {% hint style="info" %} In the following code blocks, the dollar sign `$` just means you should be typing what follows in your terminal/command prompt! {% endhint %} ## 1. Install `virtualenv` Navigate to the directory that you want to be coding in, and make sure you have `virtualenv`installed ``` $ pip install virtualenv ``` ## 2. Create a `venv` folder You could actually name this anything you want, but it's good practice to call it `venv` You create this folder so you can activate later to essentially "enter your virtual environment" ``` $ python3 -m venv ``` ## 3. Activate the `venv` folder You can see if it's been created with `ls` for MacOS or `dir` for Windows **MacOS** command to activate virtual environment: ``` $ . venv/bin/activate ``` **Windows** command to activate virtual environment: ``` $ .\venv\Scripts\Activate.ps1 ``` If you are getting an error that reads: `"cannot be loaded because the execution of scripts is disabled on this system".`, close PowerShell (if open), look up PowerShell on Windows search, click on "run as administrator", then run the command below: ``` $ Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser ``` If you are curious as to what exactly this is doing to your computer, it is allowing the current user to run scripts. `RemoteSigned` is one of many execution policies which you can read about [here](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_execution_policies?view=powershell-7.2#powershell-execution-policies), and `CurrentUser` is one of many scopes which you can read about [here](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_execution_policies?view=powershell-7.2#execution-policy-scope). If you're still getting an error, try running these commands on PowerShell instead of Command Prompt. ## 4. Install `requirements.txt` Now that you're in the environment, your terminal should have a `(venv)` next to your cursor, so you can now install the requirements to setup your virtual environment ``` (venv) $ pip3 install -r requirements.txt ``` Make sure `requirements.txt` exists where you are running this command! (again, you can check this using `ls`/`dir` to list everything in your current directory). If you are curious, `-r` is a flag that installs the requirements *recursively* to install all requirements defined in `requirements.txt`.3 ## 5. Run your code! Now you can run code with the dependencies that were installed by Step 4. Here's just an example: ``` (venv) $ python3 .py ``` ## 6. If you're done, deactivate If you're done developing for your project for the time being, or if you want to work on another project that has another virtual environment, you can deactivate the one that you're currently in ``` (venv) $ deactivate $ ``` {% hint style="info" %} ## What commands to use and when to use them * Steps **#1, 2, and 4** are only done once for every new project you'll be creating a virtual environment for. * Steps **#3, 5, and 6** are what you'll be using every time after that to use your virtual environment! {% endhint %} ## Common Errors ### no such file or directory: `requirements.txt` There are two issues that cause this problem: 1. You are not in the directory where `requirements.txt` exists. You'll know this is the case if enter `ls` (MacOS) or `dir` (Windows) into your terminal and you don't see `requirements.txt` or if you didn't spell it right in your command. 2. You are trying to do `pip3 install -r requirements.txt` inside of a correctly created and activated `venv`, but **your `venv` is nested within a path that contains a folder that has a space in the name**. Here's a good [Stack Overflow post](https://stackoverflow.com/questions/7911003/cant-install-via-pip-with-virtualenv) that explains the issue. # 1. Routes {% content-ref url="/pages/-MCc2l87wHgOUpa\_r6vS" %} [Pre-Class TODO's](/chapters/routes/pre-class) {% endcontent-ref %} {% content-ref url="/pages/-LupIDxkzRD-vPxC-YPV" %} [Lecture](/chapters/routes/lecture) {% endcontent-ref %} {% content-ref url="/pages/-Lu9\_JcULXkKzXUl17ii" %} [Assignment Handout](/chapters/routes/handout) {% endcontent-ref %} {% content-ref url="/pages/-Lu9\_ONXDoxcG-L-SC0v" %} [API Specification](/chapters/routes/api) {% endcontent-ref %} # Pre-Class TODO's ### 1. Download Postman {% embed url="" %} ### 2. Install Python 3.9.12 You can check your python installation with ```bash >>> python3 --version ``` {% hint style="info" %} You may also have access to Python 3.9.12 via the `python3` command {% endhint %} If you do not have Python 3.9.12 installed on your machine, download it [here](https://www.python.org/downloads/release/python-3912/). This is the same version that is used in CS 1110 this semester. ### 3. Download Demo Starter Code {% file src="/files/zpLDOv5JyuvNHlnNqJ2a" %} ### 4. Set up your Virtual Environment **Windows** ```bash # to create a venv in your current directory, run >>> python3 -m venv venv # to activate your venv, from the folder containing the venv folder run >>> .\venv\Scripts\Activate.ps1 # to deactivate your venv, run >>> deactivate ``` **Mac/Linux** ```bash # to create a venv in your current directory, run $ python3 -m venv venv # to activate your venv, from the directory containing the venv folder run $ source ./venv/bin/activate # to deactivate your venv, run $ deactivate ``` For a more detailed rundown (and/or) troubleshooting on virtual environments, click [here](/cheat-sheets/virtual-environment). # Lecture Contributors: Alicia Wang, Conner Swenberg ## Lecture Slides {% file src="/files/XdDKpRXOkKNjYAbqYhMl" %} ![](/files/-LwYoPoWIt4q8Qv-Z9ZN) ## Clients Clients are the computers we use everyday, including phones, tablets, laptops, desktops, and gaming consoles. Clients run code locally on their machine. Things like rendering images, reacting to button clicks and user input are all done as part of the frontend of an application. What makes an application truly “networked”, i.e. involving actions from other clients, is the backend of the application. These backends exist in the cloud, running on servers. ## Servers Servers are also computers, just like clients. Servers centralize information and run backend code to execute operations to data and communicate with clients. This is the **Server-Client Model** and it is how the internet is architected today. Independent machines (like a phone, computer, etc.) make requests to servers and use the data returned in responses. ## Requests Requests are the network calls transferred over the web. Let’s dissect a request made to `http://www.google.com`: \ `http`: Hyper Text Transfer Protocol defines the protocol for the request and is reserved for port 80 on servers. HTTP requests can be made with one of many types of **methods**. A method indicates the type of request being sent and how to handle the request. The most common methods used are GET, POST, DELETE, PUT, PATCH and are all standardized by HTTP protocol. Methods indicate the purpose of the request. GET is for information retrieval, POST for information transmission, DELETE for deleting, PUT & PATCH both for updating. Requests can also contain metadata, optional information located in the **body** of the request. Metadata can be in many forms, the two most common are XML and JSON. A request **body** is likely populating in the case of creating or updating items in the database. `www`: The subdomain we are trying to access. `google`: The domain we are trying to access. A Domain Name Service (DNS) will map this domain to a specific server for you to communicate with. Going to `http://www.google.com` makes a GET request to the google domain. Similarly we can send a POST request to `http://www.google.com/login` to login our user. In the case of logging in, our POST request’s body will likely contain email/password information to be authenticated on the server. ## Responses Servers respond the exact same way with their own network calls. They also obey HTTP protocol and have the ability to contain metadata in a response body. Responses also contain specific codes giving us a high-level understanding of how the request was handled. Some common examples are: * 200 → Successful * 404 → Not found * 500 → Internal server error (an uncaught exception) ## The Whole Process 1. A client makes a request for specific pieces of data that are sent out over the internet 2. The request is received by a server 3. The server runs internal processes to create, retrieve, update, or delete items 4. The server returns response back to client over the internet # Assignment Handout Contributors: Alicia Wang, Conner Swenberg ## Assignment Scope {% hint style="warning" %} **Due Date:** Monday 3/23, 11:59pm EST {% endhint %} You will be building a [Reddit](https://www.reddit.com/)-style community forum where people can make **posts** and make **comments** on posts. ### 1. Download Starter Code We have provided the starting files you will need to complete this assignment for your convenience. This also simplifies your submission by just having to zip this same folder. {% file src="/files/Dk4eJpGUN4uMJdRJyrrB" %} ### 2. Implement API Specification We have shown you how to use the Flask framework to set up routes and respond to network requests. Your assignment is to implement a series of routes following the provided specification for responding to sample requests. This assignment has 7 routes for you to implement. If you are struggling with the assignment, we recommend referring to the demo to see a concrete example. {% content-ref url="/pages/-Lu9\_ONXDoxcG-L-SC0v" %} [API Specification](/chapters/routes/api) {% endcontent-ref %} #### 2a. Create a global variable for storing Post data Just like we showed in the demo, we want you to create a datastore for all posts using a Python dictionary. Your keys should be integers corresponding to post ids and values should be post dictionary objects that mimic the data shown in the API Specification. You may also pre-populate your dictionary with data for testing purposes. #### 2b. Implement routes Reference the demo for specifics on defining route syntax. We recommend approaching routes in the order they appear in the API specification. After defining your route and function to return a response, open up Postman and test your route. Testing with Postman simply involves creating a new request tab, entering your server's URL, choosing a request method, and clicking the `Send` button. #### 2c. Test with provided test script {% file src="/files/YMqvsmXNCWerPwOTCfqq" %} Use the provided test script to test your implementation. To run the test script, add the pa1\_test.py file to your src directory, and run: 
>>> python3 pa1_test.py
Remember that we are checking that your code matches the API specification exactly! Some common errors that can cause the test cases to fail:
* Ports not matching (5000) * Routes misspelled * Forgetting to jsonify return ### 3. Submit Assignment #### 3a. Fill out your README.txt A `README.txt` file is included in the starter code for you to fill out after completing the assignment. Also note that you will not receive extra credit for extra credit challenges you complete if you do not let us know for when we grade! {% code title="README.txt" %} ``` Name: Jane Smith NetID: js123 Challenges Attempted: ``` {% endcode %} #### 3b. Make sure you have proper Python styling in your code Common mistakes in styling are: * NOT HAVING documentation/comments in each of your routes (a one-line comment is sufficient!) * Naming variables with CamelCase instead of snake\_case * Too much/too little empty spaces/empty lines * Leaving in commented code #### 3c. **Verify your routes using** [**Postman Tests**](https://www.youtube.com/watch?v=V2ZWdPMBwSA) **and submit an exported JSON of them** Take some time looking through the guide to learn how to make tests using Postman. You should submit your Postman collection along with your CMS submission. The best way to get full credit for your tests is to ensure that you check that the response from your API matches what the API specification is expecting to receive. Please make sure that all your tests pass when we run your collection (make sure to clear your database before running to ensure you get the result we will get). While that should be enough to get full credit, your tests can be cleaner and easier to change if you use [variables](https://learning.postman.com/docs/sending-requests/variables/) or [environments](https://learning.postman.com/docs/sending-requests/managing-environments/). It's up to you how much you want to learn and incorporate into your tests - the better you get at it the easier the assignments will be in the future and the likelier your API will pass the test cases we release on the first try. If you'd prefer to learn how to use Postman through a video, [here](https://youtu.be/V2ZWdPMBwSA) is a link to everything you'll need to know for this class; however, the audio quality is quite poor due to a technical error. If you think you would benefit from a re-recording of a Postman demo, indicate that on the [feedback form](https://docs.google.com/forms/d/e/1FAIpQLSfRXyPmpfSkEQjDHtAPqSTbBCV47cR6O0363TgbxhzUsT9WHQ/viewform?usp=sf_link) and we will be happy to record one. #### 3d. Zip and submit your assignment files Next, zip the starter folder and submit on CMS. For reference, your directory structure should look like the following: ``` pa1/ |-README.txt |-postman_collection.json |-src/ |-app.py |-requirements.txt ``` For clarification, this means that you **SHOULD NOT** include your virtual environment or pycache file in your final submission. Doing so will lose you a few points on the project. #### 3e. Submit Feedback Form To receive credit for your assignment, you **must** fill out the feedback form (link on the left sidebar of the textbook) so that we can better understand how effectively we are teaching and how long students are spending on assignments. ### 4. Prepare for Next Lecture Students will most benefit from our class time if they come in with a few things done. {% content-ref url="/pages/-M1OOdG4nVhm0\_8yzC4M" %} [Pre-Class TODO's](/chapters/databases/pre-class) {% endcontent-ref %} ## Optional Challenges ### **Tier I** **+1 point**
**Validate body content of POST requests** You've already enforced that post bodies come in with the required fields, now make sure those fields contain valid content. We will passing in malformed inputs that violate * type preconditions * logical preconditions (e.g. invalid URL\*) We are expecting a **400 Bad Request** to be thrown when validating the contents as well. To ensure that your changes do not conflict with the base test cases, please create these changes in new routes under **`POST`**`/api/extra/` Specifically, we are looking for changes in all post requests so **`POST`**`/api/extra/posts/` **`POST`**`/api/extra/posts/{id}/comments/` **`POST`**`/api/extra/posts/{pid}/comments/{cid}/` ### **Tier II** **+1 point** **Allow sorting through URL parameters** URL parameters are a way to pass data through a GET request through the request URL (just like how we pass data through a POST request by using a post body). We want you to build functionality to accept a URL parameter at the end of the `/api/posts/` route (now as `/api/extra/posts/`) in the following form: `{WHEREVER YOUR SERVER IS HOSTED}/api/extra/posts/?key1=value1&key2=value2…` Since we want to sort the most popular posts (according to number of upvotes), you will also need to implement your own route to increment the upvote value of a post. **`POST`**`/api/extra/posts/{id}/` {% code title="Request" %} ```javascript { "upvotes": 3 } ``` {% endcode %} {% code title="Success Response" %} ```javascript { "id": , "upvotes": + 3, "title": , "link": , "username": , } ``` {% endcode %} **Make the default behavior of calling this route (making a request with no post body) increment the upvotes by 1.** **You will need to Google around on your own to implement this.** We highly recommend trying to Google your question and find a StackOverflow post first (chances are someone has tried to do this before) and if you cannot find anything then perhaps consider the official [Flask documentation](https://flask.palletsprojects.com/en/1.1.x/) (though this may be very dense to look through!). Of course, feel free to ask TAs during office hours if you are really stumped! Please use the following naming conventions for your URL parameters: * Key: `sort` * Value: `increasing` or `decreasing` So, for example, `/api/extra/posts/?sort=increasing` would return posts in increasing order. ### Total Extra Credit Points to Gain: 3 # API Specification Contributors: Alicia Wang, Conner Swenberg Values wrapped in `< >` are placeholders for what the field values should be. Also be sure to read the request route carefully when you implement it - typos WILL cause test cases to fail! {% hint style="danger" %} **NOTE ABOUT ERROR RESPONSES** {% endhint %} The server should return an error response for: * `POST` requests, if the user does not supply one of the fields in the body (e.g. title, link, username, text, etc.) with a status code of 400 (bad request) * Any request if the id in the URL does not exist, with a status code of 404 (not found) {% code title="Error Response" %} ```javascript { "error": "Your error message here" } ``` {% endcode %} For each of the following routes, we will specify a success response that you must implement, but make sure to also implement error responses accordingly! ## Expected Functionality ### Get all posts **`GET`**`/api/posts/` {% code title="Success Response" %} ```javascript { "posts": [ { "id": 0, "upvotes": 1, "title": "My cat is the cutest!", "link": "https://i.imgur.com/jseZqNK.jpg", "username": "alicia98" }, { "id": 1, "upvotes": 3, "title": "Cat loaf", "link": "https://i.imgur.com/TJ46wX4.jpg", "username": "alicia98" }, ... ] } ``` {% endcode %} ### Create a post **`POST`**`/api/posts/` {% code title="Request" %} ```javascript { "title": "I love my dog!", "link": "https://i.imgur.com/XsaLqi1.jpg", "username": "raahi014" } ``` {% endcode %} {% code title="Success Response" %} ```javascript { "id": , "upvotes": 1, "title": "I love my dog!", "link": "https://i.imgur.com/XsaLqi1.jpg", "username": "raahi014" } ``` {% endcode %} ### Get a specific post **`GET`**`/api/posts/{id}/` {% code title="Success Response" %} ```javascript { "id": , "upvotes": , "title": , "link": , "username": } ``` {% endcode %} ### Delete a specific post **`DELETE`**`/api/posts/{id}/` {% code title="Success Response" %} ```javascript { "id": , "upvotes": , "title": , "link": , "username": } ``` {% endcode %} ### Get comments for a specific post **`GET`**`/api/posts/{id}/comments/` {% code title="Success Response" %} ```javascript { "comments": [ { "id": 0, "upvotes": 8, "text": "Wow, my first Reddit gold!", "username": "alicia98" }, ... ] } ``` {% endcode %} ### Post a comment for a specific post **`POST`**`/api/posts/{id}/comments/` Note: comments should have **globally** unique IDs! That is, if a comment on one post has ID 5, no other comments, *even on other posts* should have an ID of 5! {% code title="Request" %} ```javascript { "text": "what a cute puppy aww", "username": "raahi014" } ``` {% endcode %} {% code title="Success Response" %} ```javascript { "id": , "upvotes": 1, "text": "what a cute puppy aww", "username": "raahi014" } ``` {% endcode %} ### Edit a comment for a specific post **`POST`**`/api/posts/{pid}/comments/{cid}/` {% code title="Request" %} ```javascript { "text": "what a cute puppy aww (edit): omg ty for the likes" } ``` {% endcode %} {% code title="Success Response" %} ```javascript { "id": , "upvotes": <# UPVOTES>, "text": "what a cute puppy aww (edit): omg ty for the likes", "username": } ``` {% endcode %} # 2. Databases {% content-ref url="/pages/-M1OOdG4nVhm0\_8yzC4M" %} [Pre-Class TODO's](/chapters/databases/pre-class) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fKKNIsG1xQYC8sew" %} [Lecture](/chapters/databases/lecture) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fMm817ngKKGIpOIT" %} [Demo](/chapters/databases/demo) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fP7TIX6kiXkY0Pkn" %} [Assignment Handout](/chapters/databases/handout) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fS-Hc\_Ch\_pKAWu9E" %} [API Specification](/chapters/databases/api) {% endcontent-ref %} # Pre-Class TODO's Contributors: Conner Swenberg ### 1. Download Demo Starter Code {% file src="/files/XqK8ecoShz67UiBA7399" %} ### 2. Install SQLite3 We will be creating a formal database and will be using the SQLite3 package to create and manipulate our tables. Try typing `sqlite3` in your Command Line (Terminal, Powershell, etc.). If the command runs successfully, you do not need to take any further actions. Otherwise, you can download SQLite [here](https://www.sqlite.org/download.html). You should install the package under **Precompiled Binaries** for your respective operating system. #### Mac Installation For MacOS, you would download one of two packages under **Precompiled Binaries for Mac OS X**, based on your CPU architecture: `sqlite-tools-osx-arm64`, which you would use if your Mac has Apple Silicon (ARM), which is typical of most modern Macs (M1, M2, M3 Macs); `sqlite-tools-osx-x64`, which you would use if your Mac has an Intel CPU, which is typical of older Macs (pre-2020). #### Windows Installation For Windows, you would download the package `sqlite-tools-win-x64` under **Precompiled Binaries for Windows**. You can reference [this](https://www.tutorialspoint.com/sqlite/sqlite_installation.htm) for installation instructions after downloading the zip file, which will walk you through some shell commands to run. # Lecture Contributors: Alicia Wang, Conner Swenberg ## Lecture Slides {% file src="/files/dqsUizlzM3MaF6ThjBnV" %} ## Databases ### The Need for Databases Last time we setup a simple flask app that can receive requests and return responses. However, our application stores data in memory; what if we want to persist this non-static data? In any sort of meaningful application, various entities that exist within the logic of your app have relationships with other entities.\ In the example of a social network: users, posts, photos, comments are all entities that are related to each other in some way. Databases allow you to store these entities and and make said “connections” between other entities. ### What is a Database? A database is a **collection of information** organized so that it can be easily accessed, updated, and managed. This is executed with a collection of tables. Each column represents a specific piece of information you wish to store and has a defined data type. Each row contains a set of values corresponding with each column. A row represents a singular item.\ **Example: A grocery database**This database contains two tables, perhaps named fruits and employees. Both tables contain three rows (also referred to as records or tuples) and four columns (also referred to as fields or attributes). ![](https://paper-attachments.dropbox.com/s_B71A9140CF7DD42267FBEF3E5897F97C273F50122B299C1C2D40B1AABEFD2B4A_1570254521860_image.png) ### Why use a Database? **1. Querying** You can query data, i.e. ask database questions. With SQL databases, it is easy to receive data under some constraints. For example, you can easily find all items with at least 10 in stock in the database above. **2. Structure** All data in a SQL database is structured according to a pre-defined schema, and the structure will be enforced to avoid potential data mishandling. In the database above, the age column in the Employee table is an integer, so we could not create a new entry with a string in that field. SQL databases also grant the ability to relate data from two different tables together, which will be covered next lecture. ### SQL Structured Query Language: Language used to make queries that can read, insert, update or delete data from a database. SQL is our means of communicating with the database. **SQLite** SQLite is a relational database management system (RDBMS). A **RDBMS** allows you to create, update, and administer a relational database. Unlike other RDBMSs (MySQL, PostgreSQL, etc), SQLite: * Does not require installation, setup, or configuration * Is serverless * Reads and writes directly to ordinary disk files * Has databases is contained in a single file **Pros** * Doesn’t rely on a server to run * Has nothing to configure * Is small (library only 250KB) * Is file based and extremely portable **Cons** * Is only a subset of SQL * Requires a lot of memory to run for large databases * Has no user/privilege management ## SQL Operation Examples ### **Creating a new table** ```sql CREATE TABLE table_name ( column1 datatype PRIMARY KEY AUTOINCREMENT, column2 datatype NOT NULL ); ``` This query will create a new table in your database. The column parameters specify the name of each table attribute/field. The datatype parameters specify what type of data each column can hold (e.g. INT, TEXT, etc.) Columns can have special constraints placed on them. These constraints are specified after the column type, like so: ```sql columnname datatype constraint ``` Column constraints are used to limit the data that is inputted into the table, ensuring the reliability of the data. If a query that violates a constraint is made, the operation will be aborted. The `NOT NULL` constraint forces a column to not accept `NULL` values such that you cannot insert or update a row without giving a value for this column. The `PRIMARY KEY` constraint makes a column a unique identifier of a row in the table. Each value in a `PRIMARY KEY` column must be unique to the table and not `NULL`. ### **Inserting data into a table** ```sql INSERT INTO table_name (column1, ..., columnN) VALUES (value1, ..., valueN); ``` `INSERT INTO` is used to add new rows to a table. This query specifies each column and its respective value to be inserted. If you are adding values for all columns of a table, you don’t need to specify the columns as long as the values are listed in the same order as the columns ```sql INSERT INTO table_name VALUES (value1, ..., valueN); ``` ### **Retrieving data from a table** ```sql SELECT column1, ..., columnN FROM table_name; ``` `SELECT` is used to retrieve data from a table. The data returned is stored in a result-set, which looks like a table with columns that are a subset of the original table’s columns. The column parameters are the columns you want to select data from. You can also select data from all columns in a table using the `*` symbol: ```sql SELECT * FROM table_name; ``` To filter rows in a `SELECT` query, you can use the `WHERE` clause, which is used to select only those records that fulfill a specified boolean condition: ```sql SELECT column1, ..., columnN FROM table_name WHERE condition; ``` ### **Updating data in a table** ```sql UPDATE table_name SET column1=value1, ..., columnN=valueN WHERE condition; ``` `UPDATE` is used to modify existing rows in a table. It will change the columns to the value you specified for rows that meet the boolean condition. If no `WHERE` clause is included, then all rows will be updated. ### **Destroying data in a table** ```sql DELETE FROM table_name WHERE condition; ``` `DELETE FROM` will delete all rows in a table that satisfy the `WHERE` clause. ### **Case Sensitivity** SQL keywords are case insensitive, so you can use `SELECT` or select interchangeably. In SQLite, database and column names are also case insensitive, but they may be in other database management systems! # Demo Contributors: Alicia Wang, Conner Swenberg {% content-ref url="/pages/-M1OOdG4nVhm0\_8yzC4M" %} [Pre-Class TODO's](/chapters/databases/pre-class) {% endcontent-ref %} ## SQLite3 Demo {% hint style="info" %} To clear your SQLite3 shell: * for Linux/MacOS: `.shell clear` * for Windows: `.shell cls` {% endhint %} ### Create a Table Creating a table containing rows of users with an ID, name, age, and optional email. {% hint style="info" %} * No comma at the last line! * Semicolons are a must! It tells SQL that we are done with our command * Words in caps are SQL keywords, and we do this out of readability to distinguish between our actual data, like the name of our table (`user`) and the names of our columns, also known as fields, (`name`, `age`, `email`) {% endhint %} ```sql CREATE TABLE user ( id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL, age INT NOT NULL, email CHAR(50) ); ``` Creating a table containing rows of fruit with an ID, name, and whether or not it is healthy. {% hint style="info" %} * SQL has no official boolean data type, so we must use the INTEGER data type to denote 0 as False and 1 as True * Naming conventions of multiple words is with underscores, like the following example with `is_healthy` {% endhint %} ```sql CREATE TABLE fruit ( id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL, is_healthy INTEGER NOT NULL ); ``` ### See Table Columns To see that we created the columns that we intended! ```sql $ PRAGMA table_info(fruit); ``` ### See Tables To verify that our tables have actually been added to our database! {% hint style="info" %} If you exit out of your SQLite shell, and did not start by `.open filename.db` or save your session to a file, you will lose your progress, and `.tables` will be empty! {% endhint %} ```sql .tables; ``` ### **Delete a Table** Delete the fruit table. {% hint style="info" %} Check that it's been deleted with `.tables`! {% endhint %} ```sql DROP TABLE fruit; ``` ### Add Data Into a Table Add a user named `Alanna` with age `20` and email `alanna@gmail.com`. {% hint style="info" %} Hit `enter` on your keyboard to introduce new lines that help you read your SQL commands better. White spaces do not affect anything! {% endhint %} ```sql INSERT INTO user (name, age, email) VALUES ("Alanna", 20, "alanna@gmail.com"); ``` Add two more users to our database, `Conner` and `Alicia`. ```sql INSERT INTO user (name, age, email) VALUES ("Conner", 21, "conner@gmail.com"); INSERT INTO user (name, age) VALUES ("Alicia", 21); ``` {% hint style="info" %} * The `NOT NULL CONSTRAINT FAILED` error occurs if you don't insert a value into a column with the `NOT NULL` label you may have written when you defined your table * Values must match the number of columns you list and the order that you list them as well {% endhint %} ### Retrieve Data From a Table Get data from only two columns from our `user` table, `name` and `email`. ```sql SELECT name, email FROM user; ``` Get all data from our `user` table. ```sql SELECT * FROM user; ``` Get data from two columns, `name` and `age`, that satisfies the condition that the user is 18 years old or older. ```sql SELECT name, age FROM user WHERE age >= 18; ``` Use `AND` or `OR`, SQL keywords, to chain conditionals. ```sql SELECT id, name, age FROM user WHERE name = "Alicia" AND age > 20; ``` ### Update Data in a Table Replace the email for only the user with ID as 1, which happens to be `Alanna`. ```sql UPDATE user SET email = "asz33@cornell.edu" WHERE id = 1; ``` ### **Delete Data from a Table** Delete users who are 21 years old from our `user` table, which happens to be `Conner` and `Alicia`. You can verify this with `SELECT * FROM user;`. ```sql DELETE FROM user WHERE age = 21; ``` ## File Setup {% code title="db.py" %} ```python import os import json import sqlite3 class DatabaseDriver(object): """ Database driver for the Task app. Handles with reading and writing data with the database. """ def __init__(self): pass ``` {% endcode %} First we will begin by setting up a new file, `db.py`, to house our database implementation. Notice how we are importing the `sqlite3` package in line 3. Our implementation approach is to construct this `DatabaseDriver` object as our interface for manipulating a SQLite database. We will define many methods within this class to allow us to read and write to our tables and abstract the calling of these methods in `app.py`. Now back in `app.py` we want to instantiate our database driver as a variable as follows: {% code title="app.py" %} ```python import json from flask import Flask, request import db DB = db.DatabaseDriver() app = Flask(__name__) ``` {% endcode %} We first import `db.py` at the top and then instantiate an instance of our DB driver class like any other python class. We will interact with our database within `app.py` through calling methods of this object. Note: if you are continuing on the same files from last week's demo, you can remove the initialization of our `task_id_counter` and `tasks` dictionary as these are being replaced the use of a real database. ## Database methods ### Database Initialization To begin, let us complete our init function: ```python def __init__(self): self.conn = sqlite3.connect("todo.db") self.create_task_table() ``` **Connect to the Database:** Our first line that sets `self.conn` represents our connection to the database. You can imagine calling `sqlite.connect()` as analogous to opening an Excel spreadsheet, loading in the contents of our tables for reading and manipulation. We are connecting to a database called `"todo.db"`, thus implying that our database can be stored locally as a single file, which is a convenient way for us to store data for our purposes. By default, SQLite ensures that the database connection can only be used in the same thread where it was created, which helps prevent potential issues when working in multi-threaded environments. Since our course doesn't involve multithreading, we don't need to modify this behavior. **Initialize our Tables:** Our next and final step for initializing our database is to create whatever tables we want to work with. For right now, we are calling a method `create_task_table()` that we have actually not defined yes, so let's do that: ```python def create_task_table(self): try: self.conn.execute(""" CREATE TABLE task ( ID INTEGER PRIMARY KEY AUTOINCREMENT, DESCRIPTION TEXT NOT NULL, DONE BOOLEAN NOT NULL ); """) except Exception as e: print(e) ``` Firstly, we wrap our function in a try-except so that if the table already exists, we can still proceed normally. We will be using `self.conn.execute()` often throughout our implementation and the function simply allows us to execute a SQL command as provided by a python string (we are using triple quotes, `"""` to write multi-line strings). As the function name indicates, we will be creating a table to store our tasks that contains the same fields as our previous task app implementation. As noted in lecture, each column definition involves providing a field name and column type. We can also provide the `PRIMARY KEY` flag on our `ID` field which tells SQLite that this is must be unique across the table and needs to be incremented as we create new rows in the table. The `NOT NULL` flag indicates that a column cannot contain `Null` values, i.e. it must contain some real information. ### Get All Tasks Thinking back to our application's routes from `app.py` the first query we want to implement is a method to get all tasks. We can do this as follows within our DB driver class: ```python def get_all_tasks(self): cursor = self.conn.execute("SELECT * FROM task;") tasks = [] for row in cursor: tasks.append({"id": row[0], "description": row[1], "done": row[2]}) return tasks ``` **Select Data with a Cursor:** Using the same Excel table analogy, we can think of a query like dragging our cursor across many rows and columns. In our executed query, we wish to grab all columns (star `*` syntax) within the `task` table. We deliberately name this variable `cursor` to think of it like our cursor as we visually drag across the table. Executing this query will return a SQLite.Cursor object, but we can play with it like it is a 2D array (a list of lists). **Iterate over Rows:** Now to parse this information, we can loop over each row within this 2D array-like object and construct a new python object by mapping values in the row-list to their appropriate field name. Column ordering is preserved, so `row[0]` is the first column `ID`, `row[1]` the second column `DESCRIPTION`, and `row[2]` the third column `DONE`. We can construct a new list to store these new objects, which will be our final return. ### Add a Task To add new tasks to our database, we will functionally be inserting new rows into our `task` table. We can define a method that takes a `description` and `done` argument and adds a new task with these attributes to our database: ```python def insert_task_table(self, description, done): cursor = self.conn.cursor() cursor.execute("INSERT INTO task (DESCRIPTION, DONE) VALUES (?, ?);", (description, done)) self.conn.commit() return cursor.lastrowid ``` **Create a Cursor:** We can also make SQL operations by first instantiating a cursor with `self.conn.cursor()`and then executing a SQL command with this cursor. In our previous operations, we were actually taking a shortcut by writing `self.conn.execute`. which would instantiate a cursor, execute the SQL command, and return us this cursor all in one function call. **Execute the Insert Command:** To execute our insert command, we need to define the table we are inserting into, the columns we are providing values for, and our intended values. Notice that we do not define `ID` as a column we are providing a value for because one will be generated for us. In SQLite, we can mark placeholder values in our python string with `?`. We can then provide the actual values inside a tuple as a second argument for the `execute` method. **Save and Return:** Just like making a change to an Excel spreadsheet needs to be saved, so does the insert operation on our database. We save any changing to the database with the `commit` method. Upon committing our change, we will return the id of the last row available to the cursor (the task we just added). This will make more sense when we walk through how we call this method in `app.py`. ### Retrieve a Task To retrieve a specific task from our database, we will define a method that takes an `id` argument write a SQL query to execute: ```python def get_task_by_id(self, id): cursor = self.conn.execute("SELECT * FROM task WHERE ID = ?", (id,)) for row in cursor: return {"id": row[0], "description": row[1], "done": row[2]} return None ``` **Execute the Retrieval Query:** Our SQL query will select all columns (`*`) from our `task` table for rows that have their `ID` column equal to our provided ID. Notice how we use the same `?` placeholder syntax and a tuple containing our values to substitute. Note that `(id,)` defines a tuple of length one with the seemingly extraneous comma; python will not evaluate this to a tuple if the comma is missing. **Return:** Because we know that the ID field is a primary key in the `task` table, we know that this query will either return one row of data if our task exists, or none at all otherwise. Because of this we want to return on the first row loop of our cursor. If the `cursor` object contains no data, our for loop will act like it is looping over an empty list (therefore no loops) and we will return `None`. ### Update a Task To update a specific task in our database, we will utilize SQL's `UPDATE` keyword. We can define a method that takes an `id`, `description`, and `done` argument to identify our task of interest and update the fields appropriately: ```python def update_task_by_id(self, id, description, done): self.conn.execute(""" UPDATE task SET description = ?, done = ? WHERE id = ?; """, (description, done, id)) self.conn.commit() ``` **Execute the Update Command:** To translate our SQL to plain english: we wish to `UPDATE` rows within the `task` table by setting the `description` and `done` columns to our values `WHERE` the task's `id` equals our id of interest. Just like inserting a new task into the table, our update command makes a change to the database and thus needs to be committed for the changes to persist. ### Delete a Task To delete a specific task in our database, we will utilize SQL's `DELETE` keyword. We can define a method that takes an `id` argument to identify our task of interest and remove it: ```python def delete_task_by_id(self, id): self.conn.execute(""" DELETE FROM task WHERE id = ?; """, (id,)) self.conn.commit() ``` **Execute the Delete Command:** Just like previously, do not forget to commit your change to the database! ## Updates to Route Logic Now we can transition back to `app.py` to call these database methods from within our routes. But first, we will abstract our response structure to clean up our code from last time. ### Generalized Response Functions ```python app = Flask(__name__) def success_response(body, code=200): return json.dumps(body), code def failure_response(message, code=404): return json.dumps({'error': message}), code ``` As you noticed in our last demo and assignment, we use `json.dumps(...)` a lot. Our responses follow an adaptable general structure, so we can turn generating these response payloads into a function. For returning a successful response, we most often use the code `200` so we will set this as our default. Likewise, for failure responses, we most often use the code `404` because we cannot find a particular resource. ### Get All Tasks ```python @app.route("/tasks/") def get_tasks(): return success_response(DB.get_all_tasks()) ``` Retrieving all of our tasks is now as simple as calling our previously written method on our database driver. By abstracting interactions with the database to another file, we will dramatically clean up our application code. Everything else about how we define a route, construct a result dictionary, and return a JSON format of this data along with a response code remains the same from what we previously learned. ### Post a Task ```python @app.route("/tasks/", methods=["POST"]) def create_task(): body = json.loads(request.data) description = body["description"] task_id = DB.insert_task_table(description, False) task = DB.get_task_by_id(task_id) if task is None: return failure_response("Something went wrong while creating task!", 500) return success_response(task, 201) ``` Just like before, we need to parse our incoming request's body for the `description` data. We will can add a new task to the database with our `insert_task_table` method and will call `get_task_by_id` after to make sure our task was properly added. If our task was properly created, i.e. `is not None`, then we can return it and now use a response code of `201` to indicate a new resource was created. ### Retrieve a Task ```python @app.route("/tasks//") def get_task(task_id): task = DB.get_task_by_id(task_id) if task is None: return failure_response("Task not found!") return success_response(task) ``` Just like before, we design this route to accept an id within the route path. All that has changed is that we call our database method for retrieving the task, which will return `None` if there does not exist a task with the provided id. ### Update a Task ```python @app.route("/tasks//", methods=["POST"]) def update_task(task_id): body = json.loads(request.data) description = body["description"] done = bool(body["done"]) DB.update_task_by_id(task_id, description, done) task = DB.get_task_by_id(task_id) if task is None: return failure_response("Task not found!") return success_response(task) ``` Upon parsing the request's body, we will call our custom update method to make the execution. We want to return the newly updated data, so to make sure things worked properly, we retrieve our task from the database with a new query. ### Delete a Task ```python @app.route("/tasks//", methods=["DELETE"]) def delete_task(task_id): task = DB.get_task_by_id(task_id) if task is None: return failure_response("Task not found!") DB.delete_task_by_id(task_id) return success_response(task) ``` If our task does exist, we will execute the deletion, otherwise we return a failed response. # Assignment Handout Contributors: Alicia Wang, Conner Swenberg ## Assignment Scope {% hint style="warning" %} **Due Date:** Monday 4/8, 11:59pm EST {% endhint %} You will be building a Venmo-style peer-to-peer payment app. For now, this app will have users that can send money to each other. ### 1. Download Starter Code We have provided the starting files you will need to complete this assignment for your convenience. This also simplifies your submission by just having to zip this same folder. {% file src="/files/RlTaqmiFsmbfCkqmqtSJ" %} ### 2. Implement API Specification We have shown you how to use the Flask framework to set up routes and respond to network requests. Your assignment is to implement a series of routes following the provided specification for responding to sample requests. If you are struggling with the assignment, we recommend referring to the Demo to see a concrete example. {% content-ref url="/pages/-Lu9fS-Hc\_Ch\_pKAWu9E" %} [API Specification](/chapters/databases/api) {% endcontent-ref %} #### 2a. Implement DatabaseDriver class in `db.py` Reference the Demo for specifics on connecting to our Sqlite3 database file, `venmo.db`, and writing SQL command executions with the `sqlite3` package import. Implement methods for initializing the DatabaseDriver (`__init__(self)`), creating your user table, querying, inserting, updating, and deleting from the user table. Note: we have examples of SQL syntax in both the Lecture and Demo of this chapter! #### 2b. Implement routes in `app.py` Reference the Demo for specifics on defining route syntax. We recommend approaching routes in the order they appear in the API Specification. After defining your route and function to return a response, open up Postman and test your route. Testing with Postman simply involves creating a new request tab, entering your server's URL, choosing a request method, and clicking the `Send` button. #### 2c. Test your routes with testing script After creating all of your routes to satisfy API Specification (and checking with Postman as you create each route), you should run our testing script to confirm that everything is working properly. {% file src="/files/fOzVBpM353QK0064z0dc" %} For instructions on running the test script and common errors, look at the assignment handout for PA1. ### 3. Submit Assignment #### 3a. Fill out your README.txt A `README.txt` file is included in the starter code for you to fill out after completing the assignment. Also note that you will not receive extra credit for extra credit challenges you complete if you do not let us know for when we grade! {% code title="README.txt" %} ``` Name: Jane Smith NetID: js123 Challenges Attempted: ``` {% endcode %} #### 3b. Make sure you have proper Python styling in your code Common mistakes in styling are: * NOT HAVING documentation/comments in each of your methods * Naming variables with CamelCase instead of snake\_case * Too much/too little empty spaces/empty lines * Leaving in commented code #### 3c. **Verify your routes using** [**Postman Tests**](https://www.youtube.com/watch?v=V2ZWdPMBwSA) **and submit an exported JSON of them** Take some time looking through the guide to learn how to make tests using Postman. You should submit your Postman collection along with your CMS submission. The best way to get full credit for your tests is to ensure that you check that the response from your API matches what the API specification is expecting to receive. Please make sure that all your tests pass when we run your collection (make sure to clear your database before running to ensure you get the result we will get). While that should be enough to get full credit, your tests can be cleaner and easier to change if you use [variables](https://learning.postman.com/docs/sending-requests/variables/) or [environments](https://learning.postman.com/docs/sending-requests/managing-environments/). It's up to you how much you want to learn and incorporate into your tests - the better you get at it the easier the assignments will be in the future and the likelier your API will pass the test cases we release on the first try. If you'd prefer to learn how to use Postman through a video, [here](https://youtu.be/V2ZWdPMBwSA) is a link to everything you'll need to know for this class; however, the audio quality is quite poor due to a technical error. If you think you would benefit from a re-recording of a Postman demo, indicate that on the [feedback form](https://docs.google.com/forms/d/e/1FAIpQLSfRXyPmpfSkEQjDHtAPqSTbBCV47cR6O0363TgbxhzUsT9WHQ/viewform?usp=sf_link) and we will be happy to record one. #### 3d. Zip and submit your assignment files Next, zip the starter folder and submit on CMS. For reference, your directory structure should look like the following: ``` pa2/ |-README.txt |-postman_collection.json |-src/ |-app.py |-db.py |-requirements.txt ``` For clarification, this means that you **SHOULD NOT** include your virtual environment, pycache, or .db file in your final submission. Doing so will lose you a few points on the project. #### 3e. Submit Feedback Form To receive credit for your assignment, you **must** fill out the feedback form (link on the left sidebar of the textbook) so that we can better understand how effectively we are teaching and how long students are spending on assignments. ### 4. Prepare for Next Lecture Students will most benefit from our class time if they come in with a few things done. {% content-ref url="/pages/-M1U7phc16QBLEG7sbrn" %} [Pre-Class TODO's](/chapters/relational-databases/pre-class) {% endcontent-ref %} ## Optional Challenges ### **Tier I** **+0.5 points for implementation**\ **+0.5 points for README explanation** **Passwords** We don’t want users to have unauthorized access to other users' balances. Please **change how you model your database** and **add extra credit routes to support users having passwords**. Require a user’s password in the request’s body when * attempting to create a user **`POST`**`/api/extra/users/` * getting a user **`POST`**`/api/extra/user/{id}/` * sending money **`POST`**`/api/extra/send/` Include proper error handling (**401 Unauthorized**) if no password is sent or if it's incorrect. In addition, include in your `README.txt`why it is important that these routes require authorization. ### **Tier II** **+0.5 points for implementation**\ **+0.5 points for README explanation**\ **+1 point more if all of Tier I completed** **Password Hashing** You probably saved your passwords in plain text in the tier I challenge. That’s a big no-no in industry! Research what password hashing is and add the functionality to previous routes requiring a password (some suggested libraries include [hashlib](https://docs.python.org/3/library/hashlib.html)). In addition include in your `README.txt` why password hashing is important. ### **Tier III** **+0.75 points for implementation**\ **+0.75 points for environment setup**\ **+0.5 points for README explanation**\ **+1 point more if all of Tier I & II completed** **Securer Passwords** While hashing passwords is a good step towards having our passwords be stored more securely, it's not enough. Try to hash a simple password (like "*password123"*) using [this](https://appzaza.com/password-hasher) and then try to crack the password using [this](https://crackstation.net/). Do some research to answer the following questions in your `README.txt` * What are [rainbow tables](https://cyberhoot.com/cybrary/rainbow-tables/) and how do they work? * What is password salting and how can it protect against rainbow tables? * What is iterative hashing and how can it protect against rainbow tables? Once you've done your research, actually add salting and iterative hashing functionality to previous routes requiring a singly-hashed password. One thing to keep in mind is that if we were to expose our codebase to the public (like our open-source apps), our salt and number of iterations are visible and our hashed passwords are vulnerable to being cracked. We can resolve this issue by having a **secrets environment file** which will contain this sensitive information. For our project to recognize our secrets file, we need a way to load the environment file when inside our directory to access our defined variables and to unload the environment file when outside our directory to clean up our environment. To accomplish this, install [direnv](https://direnv.net/) which takes care of this for you (and can be used for every language). Once you've installed this successfully, create a file named **.env** and write the following ``` PASSWORD_SALT=??? NUMBER_OF_ITERATIONS=??? ``` where **PASSWORD\_SALT** is some text that is at least 32 characters long and **NUMBER\_OF\_ITERATIONS** is at least 500. Now, to access these environment variables you can refer to [this](https://stackoverflow.com/questions/4906977/how-to-access-environment-variable-values) in your project to retrieve the values. With your new secrets file, you can now make your code open-source (e.g. a public repository on [GitHub](https://techcrunch.com/2012/07/14/what-exactly-is-github-anyway/) which you will learn about later in the semester) since it pulls from your **.env** file which you store in a secure location! Please include your **.env** in your submission (you can trust us) so we can run your code. ### Total Points to Gain: 6 # API Specification Contributors: Alicia Wang, Conner Swenberg Values wrapped in `< >` are placeholders for what the field values should be. Also be sure to read the request route carefully when you implement it. {% hint style="danger" %} **NOTE ABOUT ERROR RESPONSES** {% endhint %} The server should return an error response for: * `POST` requests, if the user does not supply one of the fields in the body (e.g. name, username, etc.) with a status code of 400 (bad request) * `GET` , `POST` and `DELETE` requests, if the requested resource in the URL does not exist, with a status code of 404 (not found) ## Expected Functionality ### Get all users **`GET`**`/api/users/` {% code title="Response" %} ```javascript { "users": [ { "id" 1, "name": "Conner", "username": "cswenberg", }, { "id": 2, "name": "Alicia", "username": "aawang", }, ... ] } ``` {% endcode %} {% hint style="danger" %} Make sure not to include each user’s balance when getting all users - this would be a confidentiality issue in real life! {% endhint %} ### Create a user **`POST`**`/api/users/` {% code title="Request" %} ```javascript { "name": "Raahi Menon", "username": "raahi014", "balance": } ``` {% endcode %} {% code title="Response" %} ```javascript { "id": "name": "Raahi Menon", "username": "raahi014", "balance": } ``` {% endcode %} {% hint style="warning" %} If the user does not supply a name, a username, or both, you should return a failure response with a descriptive error message {% endhint %} {% hint style="info" %} You can give a default value of 0 for a key that does not exist for a dictionary like so: `body.get("balance", 0)` {% endhint %} ### Get a specific user **`GET`**`/api/user/{id}/` {% code title="Response" %} ```javascript { "id": , "name": , "username": , "balance": } ``` {% endcode %} ### Delete a specific user **`DELETE`**`/api/user/{id}/` {% code title="Response" %} ```javascript { "id": , "name": , "username": , "balance": } ``` {% endcode %} ### Send money from one user to another **`POST`**`/api/send/` {% code title="Request" %} ```javascript { "sender_id": , "receiver_id": , "amount": } ``` {% endcode %} {% code title="Response" %} ```javascript { "sender_id": , "receiver_id": , "amount": } ``` {% endcode %} {% hint style="warning" %} A sender should not be able to overdraw their balance! If a request comes in where the sender sends more money than they have, you should return an error with 400 status code. {% endhint %} {% hint style="info" %} If the client does not provide a `sender_id`, `receiver_id`, `amount`, or any combination of those, respond with a descriptive error message and a 400 status code. {% endhint %} {% hint style="info" %} When checking if the user actually sent in a `sender_id`, you should check `if sender_id is not None`, instead of `not sender_id`. This is because the number zero is a false-y value, which means `not sender_id` will return `True`, even though the user did supply an `id`, albeit a value of zero. If this is still confusing, consider using `bool()` in a Python shell: ``` >>> bool(None) False >>> bool(1) True >>> bool(0) False ``` {% endhint %} # 3. Relational Databases {% content-ref url="/pages/-M1U7phc16QBLEG7sbrn" %} [Pre-Class TODO's](/chapters/relational-databases/pre-class) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fXaQH2zB40b3m4Gs" %} [Lecture](/chapters/relational-databases/lecture) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fecva\_hXLqfOmFHv" %} [Demo](/chapters/relational-databases/demo) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fgIWTgq1yBMdVU6b" %} [Assignment Handout](/chapters/relational-databases/handout) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fit5ZR8HcdTvYp2p" %} [API Specification](/chapters/relational-databases/api) {% endcontent-ref %} # Pre-Class TODO's Contributors: Conner Swenberg, Shungo Najima ### 1. Download Demo Starter Code {% file src="/files/pEG0I09g6xp7VefQJmWw" %} # Lecture Contributors: Alicia Wang, Conner Swenberg ## Lecture Slides {% file src="/files/UYyNPDyqY9IrGwqXV45S" %} ## What is a Relational Database? So far, we have discussed databases in the context of SQL and we use SQLite in our assignments. SQL databases are also known as relational databases! A key aspect of relational databases is that they have a table-based data structure that uses a strict, predefined schema. A powerful feature of relational databases is that we can join data from different tables, which allows for the representation of relationships between different data models. **Example** Suppose we have an application with users that can create posts. A database for this application may look like this:
![](https://paper-attachments.dropbox.com/s_64ACBAB092FF5BC93C866EB28B6C224E325AFE1544E0357EF66D32B569E8C449_1572621989104_image.png) If I have a user and post table, how do I keep track of who posted what? Given a post, we may want to know who the author of the post is. We can model this by having each post refer to the id of the user that authored it, like so: ![](https://paper-attachments.dropbox.com/s_64ACBAB092FF5BC93C866EB28B6C224E325AFE1544E0357EF66D32B569E8C449_1572622139808_image.png) The addition of this user-identifying column is the basis for the formation of a relationship between users and posts. ## Relationship Types The relationship between users and posts in the previous example can be described as **one-to-many**. One user can author multiple posts, but each post can be written by only one person. There are many types of relationships between entities in the world, so there are many types of relationships we can describe with a relational database. There are three main types of relationships: 1. One-to-One 2. One-to-Many 3. Many-to-Many ### One-to-One A row x in table A is related to **only** row y in table B, and row y in table B is related **only** to row x.\ An example is Cornell students to NetIDs. Each student can only have one NetID, and each NetID can correspond only to one student. A one-to-one relationship is modeled with SQL by creating a foreign key with a unique constraint in table B that references the primary key of table A. ![Visual representation of a row in one table relating to only one row in another table](https://paper-attachments.dropbox.com/s_64ACBAB092FF5BC93C866EB28B6C224E325AFE1544E0357EF66D32B569E8C449_1572622726327_image.png) **Example** In a Slack-like application, we want to keep track of people and their profiles. One person can only have one profile, and one profile can only belong to one person. A sample database may look like this: ![](https://paper-attachments.dropbox.com/s_64ACBAB092FF5BC93C866EB28B6C224E325AFE1544E0357EF66D32B569E8C449_1572626097372_image.png) Each profile row has its own primary key id, a status, and a foreign key person\_id referencing the person that has that profile. In this example, the profile with id = 30 belongs to the person with id = 20, which is Conner. Here are the SQL commands used to generate the Person and Profile tables: ```sql CREATE TABLE person ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); CREATE TABLE profile ( id INTEGER PRIMARY KEY, status TEXT NOT NULL, person_id INTEGER UNIQUE, FOREIGN KEY person_id REFERENCES person(id) ); ``` Why is it not necessary that we have a column in the Person table specifying which profile a Person has? If we want to find the profile corresponding to Person.id = 10, we can instead make the following query: ```sql SELECT * FROM Profile WHERE person_id = 10 ``` The query will return (40, 'away', 10), which is exactly what we wanted. Thus, adding another column to Person will just result in redundant information. ### One-to-Many A row in table A can be related to **many** rows in table B, but a row in table B can only be related to **one** row in table A.
The users and posts example is a one-to-many relationship. Like the one-to-one relationship, a one-to-many relation can by modeled by using a foreign key column in table B that references a primary key in table A. However, the values in the foreign key column don’t need to be unique, as we want many rows in table B to be related to one row in table A. ![Visual representation of rows in one table relating to a single row in another table](https://paper-attachments.dropbox.com/s_64ACBAB092FF5BC93C866EB28B6C224E325AFE1544E0357EF66D32B569E8C449_1572625921835_image.png) **Example** In a Spotify-like application, we will want to keep track of albums and their songs. In this example, we’ll assume that one album can contain many songs, but one song can only belong to one album. Here is a sample database modeling this: ![](https://paper-attachments.dropbox.com/s_64ACBAB092FF5BC93C866EB28B6C224E325AFE1544E0357EF66D32B569E8C449_1572626994648_image.png) The album “IGOR” (with id = 105) has two songs: “EARFQUAKE” (id = 111) and “I THINK” (id = 112). Here are the SQL commands used to generate the Album and Song tables: ```sql CREATE TABLE Album ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); CREATE TABLE Song ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, album_id INTEGER NOT NULL, FOREIGN KEY album_id REFERENCES album(id) ); ``` Note that Song.album\_id does not have a unique constraint - this allows for the “many” part of “one-to-many”. Again, we can select all the songs belonging to a specific album through a simple query: ```sql SELECT * FROM Song WHERE album_id = 444 ``` This will return the songs “4:44” (id = 400) and “Moonlight” (id = 401) for the album “4:44” (id = 444). ### Many-to-Many **Many** rows in table A can be related to **many** rows in table B, and vice versa. More formally, row x in table A relates to many rows y1, y2, … in table B, and row y in table B relates to many rows x1, x2, … in table A. An example would be posts and hashtags in a social media platform. A user’s post can contain many hashtags (#coding, #programmerhumor), and a hashtag can be associated with many posts (“How do I exit Vim?”, “I love CS 1998!”). ![Visual representation of rows in one table relating to multiple rows in another table](https://paper-attachments.dropbox.com/s_64ACBAB092FF5BC93C866EB28B6C224E325AFE1544E0357EF66D32B569E8C449_1572627776204_image.png) **Example** Implementing a many-to-many relationship is less straightforward than the previous relationships. You may think of a few intuitive solutions, two of which are not ideal: 1. You can create a column in table A to store a list of IDs from table B and vice versa. However, deleting rows from one table can result in rows in the other table referencing non-existing rows. 2. You can create a new column for each new foreign key. This is space inefficient and may lead to many unnecessary NULL values in the added columns. 3. You can create an association table that relates the primary key of table A to the primary key of table B. This is the generally accepted solution that we will be using!\ Say that we want to model Canvas, a learning platform with students enrolled in multiple courses and courses containing multiple students. The database may look like: ![](https://paper-attachments.dropbox.com/s_64ACBAB092FF5BC93C866EB28B6C224E325AFE1544E0357EF66D32B569E8C449_1572630672400_image.png) To create a many-to-many relationship between rows in Student and rows in Course, we could create a new table with two columns: one referencing Student.id and one referencing Course.id: ![](https://paper-attachments.dropbox.com/s_64ACBAB092FF5BC93C866EB28B6C224E325AFE1544E0357EF66D32B569E8C449_1572630806252_image.png) This new table `Join`, like its name implies, serves to join together students with courses. Each row in Join represents a relation between the student with Student.id = Join.student\_id and the course with Course.id = Join.course\_id. For example, the student with id = 1 is present in two rows in Join: one with course\_id = 1110 and one with course\_id = 1998. This tells us that student abc123 is enrolled in CS1110 and CS1998. On the other side, the course with id = 1110 is present in 3 rows in Join with its associated student\_id being 1, 2, and 3 respectively. Then we know that the course CS1110 has students abc123, def456, and ghi789 enrolled in it. Here are the SQL commands used to generate Student, Join, and Course: ```sql CREATE TABLE student ( id INTEGER PRIMARY KEY, netid TEXT NOT NULL ); CREATE TABLE course ( id INTEGER PRIMARY KEY, title TEXT NOT NULL ); CREATE TABLE join ( id INTEGER PRIMARY KEY, course_id INTEGER NOT NULL, student_id INTEGER NOT NULL, FOREIGN KEY course_id REFERENCES course(id), FOREIGN KEY student_id REFERENCES student(id) ); ``` Note that Join has its own primary key id in addition to its foreign keys course\_id and student\_id. # Demo Contributors: Alicia Wang, Conner Swenberg, Alanna Zhou We will be starting where we left off in our last demo. Previously, we had defined a `task` table within our `DB` driver class and some public methods for interacting with the database. We will be demonstrating a one-to-many relationship by adding functionality for subtasks to belong under a parent task. ## Add Subtasks to Database ### Define Subtask Table One benefit of our relational modeling is that we can define a new table that relates to `task` without needing to edit our original table. We can define a new method to create our subtask table as follows: ```python def create_subtask_table(self): try: self.conn.execute(""" CREATE TABLE subtask ( ID INTEGER PRIMARY KEY, DESCRIPTION TEXT NOT NULL, DONE BOOL NOT NULL, TASK_ID INTEGER NOT NULL, FOREIGN KEY(TASK_ID) REFERENCES task(ID) ); """) except Exception as e: print(e) ``` Our subtasks will contain basic fields like `description` and `done` like our parent tasks, but what makes this table special is the addition of a foreign key. As described in lecture, we can define a new column to store the id of a task and then add this `FOREIGN KEY` constraint. To translate line 9 into plain english, we would say that we are defining our field `TASK_ID` as a `FOREIGN KEY` that `REFERENCES` the `task` table's `ID` column. To reiterate: the `ID` at the end of line 9 does not refer to the `ID` column specified in our new `subtask` table, it refers to the `ID` column in the `task` table. Just like before, we will wrap this execution within a try-except so that we can prevent ourselves from crashing if the `subtask` table already exists. ### Update Database Initialization Now we have defined a method to create our new table, we want to call this upon initializing our database driver. ```python def __init__(self): self.conn = sqlite3.connect("todo.db", check_same_thread=False) self.conn.execute("PRAGMA foreign_keys = 1") self.create_task_table() self.create_subtask_table() ``` Also notice that we have added a new SQL execution line with an unfamiliar looking command `PRAGMA`. Thinking back to our intentions with defining the foreign key relationship, our goal is to define the structure and rules for storing information in our database. We could have easily been fine just defining our `TASK_ID` column in the `subtask` table and maintain the same functionality. However, because we understand the implications of this column (i.e. that it represents a relationship between real entities), we want to enforce some restrictions on the field. In essence, our `PRAGMA` command execution on line 3 is "activating" the imposed restrictions of having a foreign key reference. Also note that we set `foreign_keys = 1` regardless of the actual *number* of foreign keys; You can think of `1` as symbolically representing `True`. By default, our `FOREIGN KEY` definition will not enforce restrictions on creating subtasks without a parent task, for example, so this line will tell SQLite to alert us of these issues as they arise. ### Add a Subtask For inserting new subtasks into our database, we will design a method do take in `description`, `done`, and `task_id` arguments and execute the typical `INSERT` SQL command: ```python def insert_subtask(self, description, done, task_id): cursor = self.conn.cusor() cursor.execute( "INSERT INTO subtask (DESCRIPTION, DONE, TASK_ID) VALUES (?, ?, ?)", (description, done, task_id) ) self.conn.commit() return cursor.lastrowid ``` This is implementation is very similar to our `insert_task` method from the last demo and don't forget to commit our changes and return the id of this newly added subtask. ### Get Subtasks of a Task To get the subtasks of a parent task, all we need is to run a SQL query on the criteria of matching `task_id` values: ```python def get_subtasks_of_task(self, task_id): cursor = self.conn.execute( "SELECT * FROM subtask WHERE TASK_ID = ?", (task_id,) ) subtasks = [] for row in cursor: subtasks.append({"id": row[0], "description": row[1], "done": bool(row[2])}) return subtasks ``` This implementation follows analogously from querying our regular tasks; all that has changed is our new `WHERE` statement to only take subtasks belonging to a particular parent. ## Add Subtasks Routes ### Create a Subtask To create a new subtask, we need to know which parent task to attach it to within the request. We will extend our typical `task/id/` route path to gain this information and append a `subtasks/` to make this path unique. Our accepted request method needs to be `POST` because we are expecting data to come in the body of the request. ```python @app.route("/task//subtasks/", methods=["POST"]) def create_subtask(task_id): body = json.loads(request.data) description = body["description"] try: subtask = { "id": Db.insert_subtask(description, False, task_id), "description": description, "done": False, "task_id": task_id } return json.dumps({"success": True, "data": subtask}) except sqlite3.IntegrityError: return json.dumps({"success": False, "error": "Task not found!'}), 404 ``` To create a new subtask, we simply call our custom method `Db.insert_subtask` and provide our desired attributes attributes (defaulting `done -> False` and `task_id -> task_id`). Just like with creating new tasks, we will nest this call within constructing a new `subtask` dictionary so that we can populate the `"id"` field. Finally, by nesting this subtask construction in a try-except, we are accounting for the case where we try to add a subtask to a task that does not exist with the given `task_id`. As noted in the `except` statement, we are expecting a `sqlite3.IntegrityError` if this case does arise. ### Get Subtasks of a Task To retrieve the subtasks of a task, we can utilize the same path as the previous method, only now allowing `GET` methods (recall that the `methods` argument is optional and defaults to `["GET"]`). ```python @app.route("/task//subtasks/") def get_subtasks_of_task(task_id): res = {"subtasks": Db.get_subtasks_of_task(task_id)} return json.dumps(res), 200 ``` Because we have abstracted all interactions with the database in `db.py`, we simply need to call our method `Db.get_subtasks_of_task(task_id)`. # Assignment Handout Contributors: Alicia Wang, Conner Swenberg ## Assignment Scope {% hint style="warning" %} **Due Date:** Monday 4/14, 11:59pm EST {% endhint %} You will be building a Venmo-style peer-to-peer payment app. You will be building an API for users to **transfer** and **request** funds (along with denying or accepting requests). ### Changes from PA2 * Introduce a model for transactions (for sending/requesting money) * `/api/send/` will be deprecated, replaced with the transaction specific routes #### The Transaction Model While sending a payment request is different from sending your own money, their relevant fields are the same. As such, we can save ourselves from having two separate tables by adding a clever field, `accepted`. This **nullable** field indicates if a sent payment request has been approved (set to `true`) or rejected (set to `false`). In the case of simply sending money, we can create a transaction with the `accepted` field preset to `true`. We can also keep track of a `timestamp` field to indicate when the transaction was last updated. Let's look at an example: {% tabs %} {% tab title="Users" %} | id | name | balance | | ------------ | ---- | ------- | | 1 | John | 20 | | 2 | Jane | 20 | | {% endtab %} | | | {% tab title="Transactions" %} | id | timestamp | sender\_id | receiver\_id | amount | accepted | | ------------- | --------- | ---------- | ------------ | ------ | -------- | | | | | | | | | {% endtab %} | | | | | | | {% endtabs %} | | | | | | Now John will **transfer** a direct payment of $5 to Jane. Because this is a **transfer**, the `accepted` field will be set to `true`. and if John has enough funds in his balance, the money will be transferred. Because Jane will be receiving money, her id will correspond with the `receiver_id` field and John with the `sender_id`. The `timestamp` field is populated with the time of creation of this payment. {% tabs %} {% tab title="Users" %} | id | name | balance | | ------------ | ---- | ------- | | 1 | John | 15 | | 2 | Jane | 25 | | {% endtab %} | | | {% tab title="Transactions" %} | id | timestamp | sender\_id | receiver\_id | amount | accepted | | ------------- | -------------------------- | ---------- | ------------ | ------ | -------- | | 1 | 2022-03-27 16:54:22.554744 | 1 | 2 | 5 | True | | {% endtab %} | | | | | | | {% endtabs %} | | | | | | Now John will send a payment **request** for $15 from Jane. Because this is a **request**, there is no change in either of their balances until Jane accepts the request. {% tabs %} {% tab title="Users" %} | id | name | balance | | ------------ | ---- | ------- | | 1 | John | 15 | | 2 | Jane | 25 | | {% endtab %} | | | {% tab title="Transactions" %} | id | timestamp | sender\_id | receiver\_id | amount | accepted | | ------------- | -------------------------- | ---------- | ------------ | ------ | -------- | | 1 | 2022-03-27 16:54:22.554744 | 1 | 2 | 5 | True | | 2 | 2022-03-27 16:55:09.324106 | 2 | 1 | 15 | Null | | {% endtab %} | | | | | | | {% endtabs %} | | | | | | Now Jane will accept the payment request and if she has enough funds, the money will be transferred. If she did not have enough money, then Jane's request to accept the request would return a failed response and the `accepted` field would still be set to `Null`. Note that the `timestamp` field is updated to the time of when it was accepted. {% tabs %} {% tab title="Users" %} | id | name | balance | | ------------ | ---- | ------- | | 1 | John | 30 | | 2 | Jane | 10 | | {% endtab %} | | | {% tab title="Transactions" %} | id | timestamp | sender\_id | receiver\_id | amount | accepted | | ------------- | -------------------------- | ---------- | ------------ | ------ | -------- | | 1 | 2022-03-27 16:54:22.554744 | 1 | 2 | 5 | True | | 2 | 2022-03-27 16:59:50.949985 | 2 | 1 | 15 | True | | {% endtab %} | | | | | | | {% endtabs %} | | | | | | Now this is just one flow of two users engaging with the application and there are many other cases to consider. As you build your own payments backend, when in doubt just think about how Venmo would handle your input. ### Additional Notes #### You will need to use multiple foreign keys for this assignment Since each transaction has a sender and receiver, you would need two foreign keys referencing the user table in this assignment. Take an example of an e-commerce database schema below: {% tabs %} {% tab title="Foreign Key Example" %} ```sql CREATE TABLE user ( id INTEGER PRIMARY KEY ); CREATE TABLE item ( id INTEGER PRIMARY KEY, buyer_id INTEGER NOT NULL, seller_id INTEGER NOT NULL, FOREIGN KEY(buyer_id) REFERENCES user(id), FOREIGN KEY(seller_id) REFERENCES user(id) ); ``` {% endtab %} {% endtabs %} #### Do not name your table `transaction` ! SQLite3 reserves the keyword `transaction` and will crash when we try to create a table with this name.We recommend a table name of `transactions`, `txn`, or `txns`. ### 1. Download Starter Code Because we are building off of the previous the assignment, we want you to reuse the code you wrote for PA2! You should keep the majority of the code written for things involving users, so you only need to add one new table for transactions and update your routes to the new API Specification. {% file src="/files/F0M3kS6TTHemvgLUlA9Y" %} ### 2. Implement API Specification We have shown you how to use the Flask framework to set up routes and respond to network requests. Your assignment is to implement a series of routes following the provided specification for responding to sample requests. This assignment has 6 routes for you to implement. If you are struggling with the assignment, we recommend referring to the Demo to see a concrete example. {% hint style="info" %} If the API Specification does not seem clear enough, our recommended approach is to just think about how the real Venmo application works and go by their standards (i.e. not allowed to send money/accept a request without sufficient funds). {% endhint %} {% content-ref url="/pages/-Lu9fit5ZR8HcdTvYp2p" %} [API Specification](/chapters/relational-databases/api) {% endcontent-ref %} #### 2b. Test your routes with testing script After creating all of your routes to satisfy API Specification (and **checking with Postman as you create each route**), you should run our testing script to confirm that everything is working properly. {% file src="/files/qt0J5veYoMJFe59De4u3" %} ### 3. Submit Assignment #### 3a. Fill out your README.txt A `README.txt` file is included in the starter code for you to fill out after completing the assignment. Also note that you will not receive extra credit for extra credit challenges you complete if you do not let us know for when we grade! {% code title="README.txt" %} ``` Name: Jane Smith NetID: js123 Challenges Attempted: ``` {% endcode %} #### 3b. Make sure you have proper Python styling in your code Common mistakes in styling are: * NOT HAVING documentation/comments in each of your methods * Naming variables with CamelCase instead of snake\_case * Too much/too little empty spaces/empty lines * Leaving in commented code #### 3c. **Verify your routes using** [**Postman Tests**](https://www.youtube.com/watch?v=V2ZWdPMBwSA) **and submit an exported JSON of them** Take some time looking through the guide to learn how to make tests using Postman. You should submit your Postman collection along with your CMS submission. The best way to get full credit for your tests is to ensure that you check that the response from your API matches what the API specification is expecting to receive. Please make sure that all your tests pass when we run your collection (make sure to clear your database before running to ensure you get the result we will get). While that should be enough to get full credit, your tests can be cleaner and easier to change if you use [variables](https://learning.postman.com/docs/sending-requests/variables/) or [environments](https://learning.postman.com/docs/sending-requests/managing-environments/). It's up to you how much you want to learn and incorporate into your tests - the better you get at it the easier the assignments will be in the future and the likelier your API will pass the test cases we release on the first try. If you'd prefer to learn how to use Postman through a video, [here](https://youtu.be/V2ZWdPMBwSA) is a link to everything you'll need to know for this class; however, the audio quality is quite poor due to a technical error. If you think you would benefit from a re-recording of a Postman demo, indicate that on the feedback form (link on the left sidebar of the textbook) and we will be happy to record one. #### 3d. Zip and submit your assignment files Next, zip the starter folder and submit on CMS. For reference, your directory structure should look like the following: ``` pa3/ |-README.txt |-postman_collection.json |-src/ |-app.py |-db.py |-requirements.txt ``` For clarification, this means that you **SHOULD NOT** include your virtual environment, pycache, or .db file in your final submission. Doing so will lose you a few points on the project. #### 3e. Submit Feedback Form To receive credit for your assignment, you **must** fill out the [feedback form](https://docs.google.com/forms/d/e/1FAIpQLSfRXyPmpfSkEQjDHtAPqSTbBCV47cR6O0363TgbxhzUsT9WHQ/viewform?usp=sf_link) so that we can better understand how effectively we are teaching and how long students are spending on assignments. ## Optional Challenges ### **Tier I** **+1 point** **Friends Forever** Allow users to friend each other and add a route for accessing all of a User’s friends: * Get a user’s friends (GET): `/api/extra/users/{id}/friends/` * Create a friend pairing between two users (POST): `/api/extra/users/{id}/friends/{id}/` Getting a user's friends should return a list of user's friends in the following format: ``` { "friends": [ , , ... ] } ``` Creating a friend pairing should return the following format: ``` ``` ### **Tier II** **+1 point**\ **+1 point more if Tier I completed** #### **Join Queries** Create a new method in `db.py` and a new route in `app.py` to execute a join query on your users and transactions to get a user given their id and all the transactions they were involved in (they can be either a sender or a receiver). * Route path: `/api/extra/users/{id}/join/` If you are stuck, refer back to demo 3 on how to use the JOIN command in SQL. Otherwise, feel free to ask questions on Ed or come to Office Hours! This route should return in the following sample format: ``` { "transactions": [ { "sender_name": "Cornell AppDev", "receiver_name": "UTea", "amount": 5, "message": "boba", "accepted": True, "timestamp": }, { "sender_name": "Cornell Engineering", "receiver_name": "Cornell AppDev", "amount": 10, "message": "Giving Day!", "accepted": True, "timestamp": }, , ... ] } ``` ### **Tier III** **+2 points**\ **+1 point more if Tier I & II completed** **Email Notifications** Venmo sends you email receipts for every transaction you execute on the platform. Make a free [Sendgrid](https://sendgrid.com/) account and use their [python package](https://github.com/sendgrid/sendgrid-python) to send email notifications to users when they receive or send money (note: this requires you to add an email field to the users table). {% hint style="warning" %} macOS: You will need to install a certificate to allow python to use SSL certificates on your computer. To do so, navigate to your python installation folder in `Applications/Python 3.x/`. Once there, double-click on the file `Install Certificates.command`. This should pull up a terminal window and execute an installation. {% endhint %} ### Total Points to Gain: **6** # API Specification Contributors: Alicia Wang, Conner Swenberg, Alanna Zhou Values wrapped in `< >` are placeholders for what the field values should be. Also be sure to read the request route carefully when you implement it. Error responses should be provided as before. ## Expected Functionality ### Get all users **`GET`**`/api/users/` {% code title="Response" %} ```javascript { "users": [ { "id" 1, "name": "Conner", "username": "cswenberg" }, { "id": 2, "name": "Alicia", "username": "aawang" }, ... ] } ``` {% endcode %} Note: Make sure not to include each user’s balance or transactions when getting all users - this would be a confidentiality issue in real life! ### Create a user **`POST`**`/api/users/` {% code title="Request" %} ```javascript { "name": "Raahi Menon", "username": "raahi014", "balance": } ``` {% endcode %} {% code title="Response" %} ```javascript { "id": "name": "Raahi Menon", "username": "raahi014", "balance": , "transactions": [] } ``` {% endcode %} ### Get a specific user **`GET`**`/api/users/{id}/` {% code title="Response" %} ```javascript { "id": , "name": , "username": , "balance": , "transactions": [ , , ... ] } ``` {% endcode %} See the below routes for what data `` should contain. ### Delete a specific user **`DELETE`**`/api/users/{id}/` {% code title="Response" %} ```javascript { "id": , "name": , "username": , "balance": , "transactions": [ , , ... ] } ``` {% endcode %} ### Create a transaction by sending or requesting money **`POST`**`/api/transactions/` {% code title="Request" %} ```javascript { "sender_id": , "receiver_id": , "amount": , "message": , "accepted": true or null // if accepted is null, record the transaction in database but don't update any balances // if accepted is true, update the balances if sender has sufficient funds } ``` {% endcode %} {% code title="Response" %} ```javascript { "id": , "timestamp": , "sender_id": , "receiver_id": , "amount": , "message": , "accepted": } ``` {% endcode %} **Note:** If the `accepted` field is true, then carry out the transaction. This means that if the transaction amount is more than the sender's balance, return an error response with status code 403. In the case of creating a payment request, the `sender_id` refers to the id of the user that will eventually send the money. Therefore, the `receiver_id` is the id of the user making the request for money. For help with `timestamp`, look into the [datetime library](https://docs.python.org/3/library/datetime.html). ### Accept or Deny a payment request **`POST`**`/api/transactions/{id}/` {% code title="Request" %} ```javascript { "accepted": true or false } ``` {% endcode %} {% code title="Response" %} ```javascript { "id": , "timestamp": , // update timestamp to time of accept/deny "sender_id": , "receiver_id": , "amount": , "message": , "accepted": } ``` {% endcode %} If `accepted` is currently `None` and the post body request's new `accepted` is `True`, check if the user has enough money in their balance. If this is satisfied, then update the transaction's `accepted` to be `True` and update the balances. If the user does not have enough money in their balance, return an error response with status code 403 (forbidden) and a relevant error message saying this. If `accepted` is currently `None` and the post body request's new `accepted` is `False`, update the transaction's `accepted` to be `False`. If `accepted` currently has some value, be it `True` or `False`, then return an error response with status code 403 (forbidden) saying that you cannot change transaction's `accepted` field if the transaction has already been accepted or denied. # 4. Abstractions {% content-ref url="/pages/-M1xJkliyd9c8vw7q8DG" %} [Pre-Class TODO's](/chapters/abstractions/pre-class) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fmEVuq40C5o\_1gnW" %} [Lecture](/chapters/abstractions/lecture) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fo8cGft84tt4ILwt" %} [Demo](/chapters/abstractions/demo) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fpn\_XYbqaje0xKLH" %} [Assignment Handout](/chapters/abstractions/handout) {% endcontent-ref %} {% content-ref url="/pages/-Luj-KDcRkUl9PvXY4R4" %} [API Specification](/chapters/abstractions/api) {% endcontent-ref %} # Pre-Class TODO's ## 1. Download Demo Starter Code {% file src="/files/ZFqjwcjM71UVzBwDj448" %} # Lecture Contributors: Alicia Wang, Conner Swenberg ## Lecture Slides {% file src="/files/62i9sWQusClKrffZPoXn" %} ## Integrating a Database We have been using raw SQL as our main method of retrieving data from the database. It has been working well for our simpler applications, but there are a few reasons why we may not want to directly make SQL queries going forward. ### **Readability** SQL, while written with the intent of being user-friendly, can have some serious readability issues. Queries can get complicated when multiple tables and sub-queries are involved. Can you immediately tell what the following query does? ```sql SELECT Country.name, T.language FROM country Country, countrylanguage CL2, (SELECT tmp.language, code FROM country, (SELECT CL.language, MAX(C.surfacearea) FROM country C, countrylanguage CL WHERE C.code=CL.countrycode GROUP BY CL.language) as tmp WHERE (surfacearea = tmp.max)) T WHERE T.code = Country.code AND T.language = CL2.language AND Country.code = CL2.countrycode; ``` The nested sub-queries, conditions, and renaming of temporary tables in complex queries are difficult for a programmer to easily read. ### **Runtime Errors** Our server application is written in Python but we are using a supporting Python package to execute raw SQL commands. If the database management system raises an exception (e.g. attempting to insert a NULL value into a NOT NULL column), our application won’t automatically handle the error and the server may crash ### **Vulnerabilities** One of the biggest drawbacks to using raw SQL in our codebase is possibility of vulnerabilities to malicious attacks. One such vulnerability is called a **SQL injection**, where a specially crafted input is used to trick SQL semantics and allow database access to an external source. The attacker may be able to query unauthorized data or corrupt or delete existing data.\ Suppose we have a login for an application. To use the application, a user needs to login with their e-mail and password.
![](https://paper-attachments.dropbox.com/s_B4B530AA1CF9F094A5A1BF5CDD7B297D847CA49763E0989BEB6388342E8C5F04_1572632345692_image.png) When the user submits their login information to the client, the client will send the data over to the backend server to check if this login is valid. The server will want to see if this email-password combination exists in the database, and may query its User table like so: ```sql SELECT * FROM User WHERE email = '$email' AND password = md5('$password'); ``` Suppose the attacker submits the following login information: * email = * password = xxx') OR 1 = 1 --] These values are inserted into the placeholder values '$email' and '$password'. Then the query will look like this with the placeholders filled in: ```sql SELECT * FROM User WHERE email = 'xxx@xxx.xxx' AND password = md5('xxx') OR 1 = 1 --]'); ``` Now the query does something completely unintended. The password field is prematurely ended with xxx') and an additional WHERE condition is added: OR 1 = 1. The double hyphen -- designates a comment in SQL, so everything after it is ignored.\ When the WHERE conditions are evaluated by SQL, the query looks like this: ```sql SELECT * FROM User WHERE FALSE AND FALSE OR TRUE; ``` The email and password conditions evaluates to FALSE as that email and password combination does not exist in the table. However, 1 = 1 will always evaluate to TRUE. Because of the additional OR clause, the entire WHERE condition will then evaluate to TRUE ```sql SELECT * FROM User WHERE TRUE; ``` SELECT will return only rows where the WHERE condition evaluates to TRUE, so in this case, all rows will be returned. The attacker now has data from the entire User table. Uh oh! ![](https://paper-attachments.dropbox.com/s_B4B530AA1CF9F094A5A1BF5CDD7B297D847CA49763E0989BEB6388342E8C5F04_1572633005689_image.png) ## Object Relational Mapping To avoid the previously mentioned issues, we can use Object Relational Mappers (ORMs) to connect to our database. You may have noticed that the relational model we use for our databases are very similar to the object-oriented paradigm we learned in CS 2110. ORMs will allow us to query and manipulate data from a database using **objects** to model the data. In effect, they represent a “database of objects” that can be used in the language of our backend (in our case, Python). We can keep data consistent between our internal server and our database. ![](https://paper-attachments.dropbox.com/s_B4B530AA1CF9F094A5A1BF5CDD7B297D847CA49763E0989BEB6388342E8C5F04_1572633347133_image.png) | **Pros of ORMs** | **Cons of ORMs** | | -------------------------------------------- | ------------------------------------ | | Readable code that resembles query reasoning | Increased amount of application code | | Same-language database and application logic | Abstracted fundamentals of SQL | | Under-the-hood connection handling | Slower query performance | # Demo Contributors: Alicia Wang, Conner Swenberg {% hint style="danger" %} This demo is not up to date with Fall 2020 as we won't be covering DAO. We recommend you watch the demo video instead. {% endhint %} This demo will introduce you to a new python package, **SQLAlchemy**, which will help us interact with the database via an object relational mapping. We will also demonstrate how to set up a `dao.py` file to interface between our application and database code. ## File Setup Write the following code at the top of our `db.py` file: {% code title="db.py" %} ```python from flask_sqlalchemy import SQLAlchemy db = SQLAlchemy() ``` {% endcode %} We first need to import our `flask_sqlalchemy` package and instantiate the SQLAlchemy class. As the name implies, db will represent our database object that we will use to read and write to our database. Moving over to `app.py`, let’s first import our necessary models from `db.py` and initialize our application. {% code title="app.py" %} ```python import json from flask import Flask, request import dao from db import db # define db filename db_filename = "todo.db" app = Flask(__name__) # setup config app.config["SQLALCHEMY_DATABASE_URI"] = f"sqlite:///{db_filename}" app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False app.config["SQLALCHEMY_ECHO"] = True # initialize app db.init_app(app) with app.app_context(): db.create_all() # generalized response formats def success_response(data, code=200): return json.dumps({"success": True, "data": data}), code def failure_response(message, code=404): return json.dumps({"success": False, "error": message}), code ``` {% endcode %} Recall that our `db` variable is an instance of the SQLAlchemy class, our database engine. To initialize our application, we will use the `init_app()` method that takes in a Flask application as an argument. Upon creating the application we will run `db.create_all()` to create all of our tables as defined by our models. ## Tasks ### Model Definition We define tables in our database just like a typical python class: {% code title="db.py" %} ```python class Task(db.Model): __tablename__ = "task" id = db.Column(db.Integer, primary_key=True) description = db.Column(db.String, nullable=False) done = db.Column(db.Boolean, nullable=False) ``` {% endcode %} #### **Class definition** For our first class, we will define the structure for our tasks. We will want to subclass off of the SQLAlchemy’s Model class to inherit all of the class properties needed to interact with this table. We also want to define the name of the table SQLAlchemy will create for us with the `__tablename__` property (note the double underscores). SQLAlchemy will autogenerate a table name for us if this property is not defined, but explicitly defining the table name can be helpful later on if we have to debug raw SQL commands and need to be sure of what tables correspond with what models. #### **Column definition** We define columns in our table via creating a new field in the class with db.Column. The column generation method takes in a mandatory argument to indicate the type of the column (all types found in [documentation](https://docs.sqlalchemy.org/en/13/core/type_basics.html)) and a series of optional arguments to define more characteristics of the column. The first column we wish to define is our unique identifier, id. We will be using an integer like before and SQLAlchemy provides a handy primary\_key=True flag to enforce the conditions we wish to set for a primary key, namely uniqueness and non-nullable. \ We define description and done with a string and boolean type respectively and also enforce that these fields are not nullable.\ \ Thinking ahead to how we will want to utilize the ORM for handling requests within our routes, we will need to convert an instance of the Task class into basic types. As mentioned prior, we need to send a standardized data type across HTTP requests (our particular choice is JSON), so let us define a method that helps us convert an object into a python dictionary. ```python def serialize(self): return { "id": self.id, "description": self.description, "done": self.done, } ``` This process of converting custom object types into the basic types is known as **serialization**. Our serialize method simply constructs a python dictionary from our object’s field values. ### 1. Get all tasks Our first method is quite basic but introduces us to the first use of our ORM. {% code title="app.py" %} ```python @app.route("/tasks/") def get_tasks(): tasks = [t.serialize() for t in Task.query.all()] return success_response(tasks) ``` {% endcode %} **Querying the database** Our first change is how we query all of our tasks. Before we had a method that used SQLite3 to write direct SQL queries on our database (in this case `SELECT * FROM task`). Now the ORM provides us a structure to query without directly writing any SQL! The `Task` class imported at the top gives us direct access to the `task` table. SQLAlchemy's `.query` reference enables us to write queries on our table and is a field for every SQLAlchemy defined class we will use. The `.all()` method functions exactly like a `SELECT * FROM task` by imposing no condition on which tasks are returned. **Constructing a return** Now because our task query call returns us **Task objects**, we need to serialize these into python dictionaries so that they are convertible to a JSON format. For this method and all others defined in the DAO, we want to provide `app.py` data that is as ready to be sent back into a response as possible which is why we will serialize it here. {% hint style="info" %} This one-line style of looping through an existing list is known as [**list comprehension**](https://www.pythonforbeginners.com/basics/list-comprehensions-in-python) and is highly recommended for simple loops like this. {% endhint %} ### 2. Create a task ```python @app.route("/tasks/", methods=["POST"]) def create_task(): body = json.loads(request.data) new_task = Task( description=body.get('description'), done=body.get('done', False) ) db.session.add(new_task) db.session.commit() return success_response(new_task.serialize(), 201) ``` **Construct Task object** First, we want to construct a python object to represent our task. Just like before, we need to parse our incoming request's body for the `description` data. We use python's built-in `.get()` method to safely parse the `body` dictionary. If there does not exist a value with the provided key, the method with return `None` by default. We can indicate a default value of `False` for the `done` variable using a second argument to `body.get()`. Next we need to use the ORM to convert this object into a real entry in the database. **Saving to the database** Note that we now have a `db` object which isolates the interaction with our database. Our `db` object has a field called `session` to keep track of our current connection to the database (remember how with sqlite3 we used `self.conn`). Just like with our sqlite3 implementation where we had separate commands for executing the insertion of a new task and the actual saving, or "**committing**", of a task, the same applies with our ORM. By keeping the addition of new entries to the database and actual committing, we free ourselves to potentially add multiple entries to different tables before accepting the cost of a database save. Now we can simply return a serialized task back to the client with a `201` response code to indicate the successful creation of a new entry. ### 3. Retrieve a task ```python @app.route("/tasks//") def get_task(task_id): task = Task.query.filter_by(id=task_id).first() if task is None: return failure_response("Task not found!") return success_response(task.serialize()) ``` Just like before, we design this route to accept an id within the route path. Now we can make use of SQLAlchemy's `filter_by` method to search for a task with the `id` field matching our provided `task_id`. This query will return all tasks that abide this criteria, functionally returning a **list** of tasks. However, we only want a single task so we call `.first()` at the end of our query to return the first instance returned in this query. This query will return `None` if there does not exist any tasks belonging to `task_id`. We return a failure response if the task does not exist in our table and otherwise return a success response with a serialized task. ### 4. Update a task ```python @app.route("/tasks//", methods=["POST"]) def update_task(task_id): task = Task.query.filter_by(id=task_id).first() if task is None: return failure_response("Task not found!") body = json.loads(request.data) task.description = body.get('description', task.description) task.done = body.get('done', task.done) db.session.commit() return success_response(task.serialize()) ``` First, we want to query for our task just like before and return if there does not exist said task. After parsing the request's body, we can update our `task` object's fields like a normal python custom object with direct assignment to the fields `task.description` and `task.done`. We get a bit fancy with our use of `.get()` by placing the current `description` or `done` value as the default if no new value is provided in the request. Therefore, absent fields in the body will not change our database. Upon assigning new values to the task, we need to commit these changes and return our serialized object. ### 5. Delete a task ```python @app.route("/tasks//", methods=["DELETE"]) def delete_task(task_id): task = Task.query.filter_by(id=task_id).first() if task is None: return failure_response("Task not found!") db.session.delete(task) db.session.commit() return success_response(task.serialize()) ``` First, we want to query for our task just like before and return if there does not exist said task. We can use SQLAlchemy's provided `delete()` method to remove our task from the database. Upon committing our changes, we can return the original task to give back to the client. ## Subtasks (One-to-Many Relationship) ### Model Definition Consider that we want to add smaller “subtasks” within each task to break it down to more manageable components. We can begin defining a Subtask model with a One-to-Many relationship with Task as follows: ```python class Subtask(db.Model): __tablename__ = "subtask" id = db.Column(db.Integer, primary_key=True) description = db.Column(db.String, nullable=False) done = db.Column(db.Boolean, nullable=False) task_id = db.Column(db.Integer, db.ForeignKey("task.id"), nullable=False) ``` We follow our same guidelines of defining a `__tablename__` field and all of our columns. But to create the relationship between a Task and Subtask, we do something special. We define task\_id as a column in the table and give it the type of an integer, but we also add the argument `db.ForeignKey("task.id")`. This part indicates the declaration of a foreign key that references the id field in the task table. To generalize this structure, we define foreign keys with: `db.ForeignKey(".")`. **Serialization** Our next step is to define how we wish to serialize a Subtask and how this serialization will interact with our existing serialization for Task. We can first write a straightforward serialization of a Subtask: ```python def serialize(self): return { "id": self.id, "description": self.description, "done": self.done } ``` #### ### Update Task Model The second part to defining a one-to-many relationship involves adding some indicator code to the model on the other side of the relationship. Within our Task model we now write: ```python class Task(db.Model): __tablename__ = "task" id = db.Column(db.Integer, primary_key=True) description = db.Column(db.String, nullable=False) done = db.Column(db.Boolean, nullable=False) # defining the reverse side of the relationship subtasks = db.relationship("Subtask", cascade="delete") ``` We create a subtasks field that allows us to query from the reverse side of the relationship by defining a db.relationship type. The first argument defines the name of the model class we are referencing (notice how this is `Subtask` instead the `subtask` table name). Our second and optional argument, cascade='delete', defines a protocol to handle a deletion of a Task object from the database. Consider we remove a Task with many Subtasks, all of these subtasks now have no parent Task to reference. By default, their `task_id` field will be set to None, but in the context of our app, we only want Subtasks to exist as nested beneath a main Task. The cascade protocol says to the ORM to delete all related Subtasks upon deleting a Task to preserve this nature of the application. #### Update Task Serializer Now let's consider how we want to access these subtasks from the client’s perspective. We have two options: add a new route for the client to request for the subtasks of a particular task **or** find a way to return subtask information in the original requests that retrieve Tasks. We will go with the latter due to its simplicity for us and the client. Let us update our Task serializer to include a new field:
```python def serialize(self): return { "id": self.id, "description": self.description, "done": self.done, # serialize each subtask related to this task "subtasks": [s.serialize() for s in self.subtasks], } ``` We can use the list comprehension approach as before to serialize all of our subtasks. Our original list, self.subtasks, utilizes our defined relationship to actually make a query and gather our related subtasks. We can then serialize each of these objects in our Task serializer method. ### 6. Create a subtask {% code title="app.py" %} ```python @app.route("/tasks//subtasks/", methods=["POST"]) def create_subtask(task_id): task = Task.query.filter_by(id=task_id).first() if task is None: return failure_response("Task not found!") body = json.loads(request.data) new_subtask = Subtask( description=body.get('description'), done=body.get('done'), task_id=task_id ) db.session.add(new_subtask) db.session.commit() return success_response(new_subtask.serialize()) ``` {% endcode %} We will create a new subtask by instantiating the `Subtask` class, just like we did for regular tasks. To complete the addition, we add the newly created subtask to our database, commit the changes and finally return the subtask serialized. Upon updating our models' structure, we can implement a new route to add subtasks to a parent task: Route Definition: When creating new subtasks, we want to provide clients with a means to indicate which task to attach it with. Of the many ways for us to do this, we will simply extend our traditional `task/task_id/` route structure by adding a `subtask/` to the route path. Because we are expecting data to come in our request, we want to accept the `POST` request method. Subtask Addition: First, we need to retrieve the task we intend to add this new subtask under and return a failed response if we cannot find it. Then, we can move forward with parsing our request's body, calling our DAO, and returning a success response. ## Categories (Many-to-Many Relationship) ### Model Definition We will demonstrate a many-to-many relationship with a categories feature. Any given task can belong to many different category groups and a category can contain many tasks associated with it. We can define a basic implementation as follows: {% code title="db.py" %} ```python class Category(db.Model): __tablename__ = "category" id = db.Column(db.Integer, primary_key=True) description = db.Column(db.String, nullable=False) color = db.Column(db.String, nullable=False) ``` {% endcode %} **Serialization** Our next step is to define the serialization of a `Category` with an analogous implementation to our previous models: ```python def serialize(self): return { "id": self.id, "description": self.description, "color": self.color } ``` ### Create Association Table As of right now, our `Category` model is not related to anything. Recall from our Relational Databases lecture that we implement a many-to-many relationship via a "join" or "association" table to connect the foreign key of one table to the foreign key another table. At the top of our `db.py` file, we can create a new variable to construct a table using `db.Table`: ```python association_table = db.Table("association", db.Model.metadata, db.Column("task_id", db.Integer, db.ForeignKey("task.id")), db.Column("category_id", db.Integer, db.ForeignKey("category.id")) ) ``` The first argument provided is the table name, which we will call `"association"`. Next we need to provide the metadata of the `Model` class. All following arguments will be the columns of our table. Our first column we will store the ids of tasks, `db.ForeignKey` type, and our second column will store the ids of categories, also `db.ForeignKey` type. ### Add Relationship Fields Now to implement this relationship, we connect our `Category` model to our association table with a `db.relationship` field: ```python class Category(db.Model): __tablename__ = "category" id = db.Column(db.Integer, primary_key=True) description = db.Column(db.String, nullable=False) color = db.Column(db.String, nullable=False) # define many-to-many relationship by connecting to association table tasks = db.relationship("Task", secondary=association_table, back_populates='categories') ``` The first argument indicates the **class name** of the model we are relating too, not the table name which is why `"Task"` is capitalized. Our second argument, `secondary`, indicates the join table to use. Finally, `back_populates` indicates to SQLAlchemy the field name on the reverse side of the relationship that will be referencing this relationship. We can see this in our addition to the `Task` model: ```python class Task(db.Model): __tablename__ = "task" id = db.Column(db.Integer, primary_key=True) description = db.Column(db.String, nullable=False) done = db.Column(db.Boolean, nullable=False) # defining the reverse side of the many-to-one relationship subtasks = db.relationship("Subtask", cascade="delete") # defining the reverse side of the many-to-many relationship, connecting to the same join table categories = db.relationship("Category", secondary=association_table, back_populates="tasks") ``` We define the new field `categories` to match the input of `back_populates` from the relationship definition in the `Category` model. We also set the `back_populates` on this side to the field name we previously defined: `"tasks"`. Our first argument, the class name, will refer to the `Category` model, and we will refer to the same `association_table`. **It is crucial that the `back_populates` tags match exactly with the field names across the two models.** #### Update Task Serializer Just like with subtasks, we want to serialize categories within a task. We can do this analogously with another new line in our `Task` serializer: ```python def serialize(self): return { "id": self.id, "description": self.description, "done": self.done, # serialize each subtask related to this task "subtasks": [s.serialize() for s in self.subtasks], # serialize each category related to this task "categories": [c.serialize() for c in self.categories] } ``` ### 7. Assign category to task ```python def assign_category(task_id, description, color): task = get_task_by_id(task_id) if task is None: return None category = _get_or_create_category(description, color) task.categories.append(category) db.session.commit() return task.serialize() ``` First, we need to retrieve the task we intend to link to our category and return `None` if we cannot find it so our route knows to return a failure response. We then can call our custom method to get or create a category from the provided description. To relate this category to our task, we can append it to the `task.categories` field. Relationship fields in SQLAlchemy can be manipulated as lists which is why we use the `.append()` method to add to our relationship. Doing so will automatically update a new entry in the previously defined association table between the two models. Now we can commit changes and return the serialized task. The request body is expected to contain a `description` field and we will try to find any categories that match this description (we will not be allowing categories of the same description to exist). If there does not exist a category with said description, we will create a new `Category` object with the description and color from the request's body. This method is **private** and only intended to be used within other DAO methods (for us, our next method to assign categories). Private methods in python are indicated with a prepended underscore `_`. Just like with subtasks, we want to add a new route for assigning a category to a task: ```python @app.route("/tasks//category/", methods=["POST"]) def assign_category(task_id): body = json.loads(request.data) task = dao.assign_category(task_id, body.get("description"), body.get("color"),) if task is None: return failure_response("Task not found!") return success_response(task) ``` Following previous design patterns, we will extend our usual path for a specific task with the `category/` string and make this route accept the `POST` request method. Because we off-loaded the important logic to our DAO, our route method remains relatively simple and clean to read. # Assignment Handout Contributors: Alicia Wang, Conner Swenberg ## Assignment Scope {% hint style="warning" %} **Due Date:** Monday 4/21, 11:59pm {% endhint %} You will be building a CMS-style classroom manager. You will need to assign **instructors** to **courses** to manage **assignments** as well as **students**. ### 1. Download Starter Code We have provided the starting files you will need to complete this assignment for your convenience. This also simplifies your submission by just having to zip this same folder. {% file src="/files/3Ho1avyjAN4FsT51r8FG" %} ### 2. Implement API Specification 2a. We have shown you how to use the Flask framework to set up routes and respond to network requests. Your assignment is to implement a series of routes following the provided specification for responding to sample requests. This assignment has 7 routes for you to implement. If you are struggling with the assignment, we recommend referring to the Demo to see a concrete example. {% content-ref url="/pages/-Luj-KDcRkUl9PvXY4R4" %} [API Specification](/chapters/abstractions/api) {% endcontent-ref %} #### 2b. Test your routes with testing script After creating all of your routes to satisfy API Specification (and **checking with Postman as you create each route**), you should run our testing script to confirm that everything is working properly. {% file src="/files/Tn4y8KHT9Z9nKcutkpXp" %} ### 3. Submit Assignment #### 3a. Fill out your README.txt A `README.txt` file is included in the starter code for you to fill out after completing the assignment. Also note that you will not receive extra credit for extra credit challenges you complete if you do not let us know for when we grade! {% code title="README.txt" %} ``` Name: Jane Smith NetID: js123 Challenges Attempted: ``` {% endcode %} #### 3b. Make sure you have proper Python styling in your code Common mistakes in styling are: * NOT HAVING documentation/comments in each of your methods * Naming variables with CamelCase instead of snake\_case * Too much/too little empty spaces/empty lines * Leaving in commented code #### 3c. **Verify your routes using** [**Postman Tests**](https://www.youtube.com/watch?v=V2ZWdPMBwSA) **and submit an exported JSON of them** Take some time looking through the guide to learn how to make tests using Postman. You should submit your Postman collection along with your CMS submission. The best way to get full credit for your tests is to ensure that you check that the response from your API matches what the API specification is expecting to receive. Please make sure that all your tests pass when we run your collection (make sure to clear your database before running to ensure you get the result we will get). While that should be enough to get full credit, your tests can be cleaner and easier to change if you use [variables](https://learning.postman.com/docs/sending-requests/variables/) or [environments](https://learning.postman.com/docs/sending-requests/managing-environments/). It's up to you how much you want to learn and incorporate into your tests - the better you get at it the easier the assignments will be in the future and the likelier your API will pass the test cases we release on the first try. If you'd prefer to learn how to use Postman through a video, [here](https://youtu.be/V2ZWdPMBwSA) is a link to everything you'll need to know for this class; however, the audio quality is quite poor due to a technical error. If you think you would benefit from a re-recording of a Postman demo, indicate that on the [feedback form](https://docs.google.com/forms/d/e/1FAIpQLSfRXyPmpfSkEQjDHtAPqSTbBCV47cR6O0363TgbxhzUsT9WHQ/viewform?usp=sf_link) and we will be happy to record one. #### 3d. Zip and submit your assignment files Next, zip the starter folder and submit on CMS. For reference, your directory structure should look like the following: ``` pa4/ |-README.txt |-postman_collection.json |-src/ |-app.py |-db.py |-requirements.txt ``` For clarification, this means that you **SHOULD NOT** include your virtual environment, pycache, or .db file in your final submission. Doing so will lose you a few points on the project. #### 3e. Submit Feedback Form To receive credit for your assignment, you **must** fill out the feedback form (link on the left sidebar of the textbook) so that we can better understand how effectively we are teaching and how long students are spending on assignments. ## Optional Challenges ### **Tier I** **+1 point** **Additional Routes** Add the following routes: Drop a student from a course: `/api/courses/{course_id}/drop/` * Post body: `{"user_id": }` * Return: * `{"error": "Course not found"}` * `{"error": "User not found"}` * `{"error": "User has not been added to this course"}` * `{}` Update an assignment: `/api/assignments/{assignment_id}/` * Post body: new values for `title`, `due_date`, or both * Return: * `{"error": "Assignment not found"}` * `{}` ### **Tier II** **+1 points**\ **+1 point more if Tier I completed** **Grading Submissions** Add a Submission model and functionality to allow instructors to assign numeric grades to students’ submissions. Note: For now, the content of the submission is arbitrary and you can just use a text field "content". This includes the following routes: Submit an assignment: `/api/assignments/{assignment_id}/submit/` * Post Body: `{"user_id": , "content": }` * Return: * `{"error": "Assignment not found"}` * `{"error": "User not found"}` * `{"error": "User does not have this assignment"}` * `{}` Grade an assignment: `/api/assignments/{assignment_id}/grade/` * Post Body: `{"submission_id": , "score": }` * Return: * `{"error": "Assignment not found"}` * `{"error": "Submission not found"}` * `{"error": "Submission does not match this assignment"}` * `{}` ### **Tier III** **+2 points**\ **+1 point more if Tier I & II completed** **Submission File Uploads** Extend your submission functionality with real file uploads. Set up an Amazon Web Services account and connect to the S3 service using the [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html) package. You will need to learn how to set up your own AWS account, and successfully connect to S3 buckets. **Recommended approach:** Use the same `submit/` route as in Tier II, but change your request body type in postman from raw `JSON` to `form-data`. The `form-data` body type allows you to send a combination of files and text data in the same request. Upon receiving the request, parse the uploaded file from the request and send them as a payload to S3 using `boto3`. The `boto3` methods will return you a string URL of where the file was stored. Save this URL in the `content` field of your submission. You can test if the file upload worked correctly by copying this link into your browser → the file should download and appear. ### Total Points to Gain: **6** # API Specification Contributors: Alicia Wang, Conner Swenberg Values wrapped in `< >` are placeholders for what the field values should be. Also be sure to read the request route carefully when you implement it. ## Expected Functionality ### Get all courses **`GET`**`/api/courses/` {% code title="Response" %} ```javascript { "courses": [ { "id": 1, "code": "CS 1998", "name": "Intro to Backend Development", "assignments": [ , ... ], "instructors": [ , ... ], "students": [ , ... ] }, { "id": 2, "code": "CS 1110", "name": "Intro to Computer Science: Python", "assignments": [ , ... ], "instructors": [ , ... ], "students": [ , ... ] } ... ] } ``` {% endcode %} {% hint style="warning" %} It is important that you return the serialized relationships WITHOUT the course field. If you attempt to return them with the course field still there, your code will end up in an infinite loop. This is because your course, when returning its relationships (say, instructors), asks the instructors which courses they teach, which will then cause those courses to return their instructors, etc. {% endhint %} ### Create a course **`POST`**`/api/courses/` {% code title="Request" %} ```javascript { "code": "CS1998", "name": "Intro to Backend Development" } ``` {% endcode %} {% code title="Response" %} ```javascript { "id": , "code": "CS1998", "name": "Intro to Backend Development", "assignments": [], "instructors": [], "students": [] } ``` {% endcode %} {% hint style="info" %} If the client does not provide a code or name or both, respond with an error message with status code 400. {% endhint %} ### Get a specific course **`GET`**`/api/courses/{id}/` {% code title="Response" %} ```javascript { "id": , "code": , "name": , "assignments": [ , ... ], "instructors": [ , ... ], "students": [ , ... ] } ``` {% endcode %} ### Delete a specific course **`DELETE`**`/api/courses/{id}/` {% code title="Response" %} ```javascript { "id": , "code": , "name": , "assignments": [ , ... ], "instructors": [ , ... ], "students": [ , ... ] } ``` {% endcode %} ### Create a user **`POST`**`/api/users/` {% code title="Request" %} ```javascript { "name": "Raahi Menon", "netid": "rm834" } ``` {% endcode %} {% code title="Response" %} ```javascript { "id": , "name": "Raahi Menon", "netid": "rm834", "courses": [] } ``` {% endcode %} {% hint style="info" %} If the client does not provide a name or netid or both, respond with an error message. {% endhint %} ### Get a specific user **`GET`**`/api/users/{id}/` {% code title="Response" %} ```javascript { "id": , "name": , "netid": , "courses": [ , ... ] } ``` {% endcode %} {% hint style="info" %} The `courses` field should contain ALL courses that user is in, both as a student and as an instructor (remember users can be both! Just like your backend instructors!). {% endhint %} ### Add a user to a course **`POST`**`/api/courses/{id}/add/` {% code title="Request" %} ```javascript { "user_id": , "type": "student" or "instructor" } ``` {% endcode %} {% code title="Response" %} ```javascript { "id": , "code": , "name": , "assignments": [ , ... ], "instructors": [ , ... ], "students": [ , ... ] } ``` {% endcode %} {% hint style="info" %} The field `instructors` or `students` in the response should include the newly added user in the respective array. {% endhint %} ### Create an assignment for a course **`POST`**`/api/courses/{id}/assignment/` {% code title="Request" %} ```javascript { "title": "PA4", "due_date": 1553354209 // in Unix time (seconds since the epoch) } ``` {% endcode %} {% code title="Response" %} ```javascript { "id": , "title": "PA4", "due_date": 1553354209, // in Unix time "course": { "id": {id}, "code": , "name": } } ``` {% endcode %} {% hint style="info" %} If the client does not provide a title or due date or both, respond with an error message (status code 400). {% endhint %} ### # 5. Containerization {% content-ref url="/pages/-M1xKLqBLagJWVIE0hfF" %} [Pre-Class TODO's](/chapters/containerization/pre-class) {% endcontent-ref %} {% content-ref url="/pages/-LvxY6HCWvz63OwVp287" %} [Docker Installation](/chapters/containerization/docker-installation) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fsxSacpkIt9c5b9w" %} [Lecture](/chapters/containerization/lecture) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fv8-6IOcOMcnDe2J" %} [Demo](/chapters/containerization/demo) {% endcontent-ref %} {% content-ref url="/pages/-Lu9fxWQbjtl63zg7s2I" %} [Assignment Handout](/chapters/containerization/handout) {% endcontent-ref %} # Pre-Class TODO's ### 1. Install Docker & Docker Compose We will be using a new tool for packaging our code in preparation for running it on a cloud server! {% content-ref url="/pages/-LvxY6HCWvz63OwVp287" %} [Docker Installation](/chapters/containerization/docker-installation) {% endcontent-ref %} ### 2. Make an account on [Docker Hub](https://hub.docker.com/). This is so that you can push your Docker images to a public place that your remote server can pull from. ### 3. Download Demo Starter Code Same as demo 4 starter code! You can reuse what we did in demo4, or you can fill in our starter code with a dummy endpoint that returns "Hello World" :) {% file src="/files/ZFqjwcjM71UVzBwDj448" %} # Docker Installation Contributors: Alicia Wang, Alanna Zhou We’ll be using Docker in this course as a platform-agnostic way of ensuring that all development is done on the same operating system, letting us (the course staff) take care of installation issues, and avoid OS-specific installation issues. ## Installation Instructions ### Mac #### 1) Install Docker {% embed url="" %} ### Windows [Docker on Windows](https://docs.docker.com/docker-for-windows/install/) works fairly well on newer machines. You may have to install the [Linux kernel update package](https://docs.microsoft.com/windows/wsl/wsl2-kernel). Try this and if it doesn't work, come to Office Hours, because the other option is daunting. #### 1) Install Docker {% embed url="" %} ### Windows (Really Bad Alternative) The other option is downloading a [Virtualbox](https://www.virtualbox.org/) (a Virtual Machine) and installing Docker through there. #### 1) Set up your VM {% embed url="" %} #### 2) Install Docker on your VM {% embed url="" %} #### 2) Install Docker Compose on your VM {% embed url="" %} ### Linux #### 1) Install Docker (Ubuntu) {% embed url="" %} #### 2) Install Docker Compose (Ubuntu) {% embed url="" %} #### For other Linux distributions, see the individual guides: {% embed url="" %} ## Is It Working? You can check that you've successfully installed Docker with: `docker --version` ## Typical Errors ### Is Docker daemon running? If you're using Docker Desktop, make sure to **start the Docker application before attempting to run** Docker commands. ### Has Docker daemon been "starting" for a long time, and unable to run? `Error response from daemon: dial unix docker.raw.sock: connect: no such file or directory` This means that your Docker desktop is having trouble running (it may be "starting" for a long time), and that might be because you don't have enough space on your computer -- try to delete some stuff! ### Is the port already being used somewhere? Try `docker rm -fv $(docker ps -aq)` but it's **recommended** that you use the Docker Compose command (that is, if you've installed Docker Compose and have a `docker-compose.yml` file in the directory that you will be running the following command): **`docker-compose down`** # Lecture Contributors: Alicia Wang, Conner Swenberg, Alanna Zhou ## Lecture Slides {% file src="/files/6iBSiuKr3s7u03lbDpq4" %} ## How do other people use our application? As we learned in the first lecture, requests are made to servers. For our application to be networked, we too need to get our code running on a publicly accessible server. This process of publishing code into a real, running application is called **deployment**. Outline of deployment process: 1. Compress code into a production environment 2. Prepare production environment to run-ready 3. Spin up server(s) 4. Download & run prepared production environment on server(s) The topic of **Containerization** covers the first two steps in this process. However before we begin, there is one thing we need to cover first. ### Environment Variables Now that we are considering running our application code on an external computer, it is essential to have a secure way to share private information. We do this through what are called **environment variables**. There are many instances in which we need to use sensitive information in our application code such as username/passwords or API secret keys. However, if our code is open source (and even when it's not!), we wouldn't want to hardcode the data in for the world to see. A solution is to use environment variables, that is to define our secret variables in an "environment file" locally (typically suffixed with `.env`). Then instead of the secret API key existing within our source code, we can simply refer to the variable containing the secret key in another file. {% code title="secrets.env" %} ``` export API_SECRET_KEY="asdfjkl" ``` {% endcode %} To allow our application deployed on the external server access to these environment variables, we can similarly create a `.env` file on the server while SSHed in. This is especially useful if our deployed server should be using different secrets than our development server. ## 1. Compressing Code Into a Production Environment ### What is Docker? Docker is a tool that allows developers to package code into a standardized unit of software called a **container**. A container contains not only your application’s source code, but also all of its dependencies so the application can run quickly and reliably from one computing environment to another. That means your application’s behavior will be consistent whether you run it on a Windows or Linux environment, or on your local computer or a remote server.\ Docker containers are similar to virtual machines in that they both isolate resources, but containers are more portable and lightweight because they virtualize the operating system instead of the hardware. Multiple Docker containers (apps) can be run on the same machine with other containers as isolated processes, but still share the same OS kernel. \ ![](https://paper-attachments.dropbox.com/s_AAA2D74DE124819999BBD32958D33EA7C14D965F2ADB8F30C5EC22BA3B0414F2_1573843880551_image.png)![](https://paper-attachments.dropbox.com/s_AAA2D74DE124819999BBD32958D33EA7C14D965F2ADB8F30C5EC22BA3B0414F2_1573843908744_image.png) To Dockerize an application, we build **images** from our source code and then run the **containers** created from the images. We can think of images and containers like classes and objects or blueprints and products. We need to specify a Dockerfile, which is the blueprint on how to use our source code and its dependencies to create an image that will run our application. ### Docker Images Docker is all about its images! A Docker image is a blueprint that is used to run code in a Docker container. In other words, it is built from instructions (that you as a developer specify, called a `Dockerfile`) for a complete and executable version of your application. An image includes elements that are needed to run an application as a container: * code * config files * environment variables * libraries Once an image is deployed to a Docker environment, it can then be executed as a Docker container. #### What does image mean? You can think of a Docker image as a "snapshot" in other virtual machine environments, a record of a Docker virtual machine, or Docker container, at a point in time. If you think of a Docker image as a digital picture, then a Docker container can be seen as a printout of that picture. #### Images cannot be changed Docker images have the special characteristic of being immutable. They can't be modified, but they can be duplicated and shared or deleted. The immutability is useful when testing new software or configurations because no matter what happens, the image will still be there, as usable as ever. So if you need to "modify" an image, you simply just make your changes and build another image. You can name it the same name, but you're essentially overwriting, not modifying. ## 2. Prepare Production Environment to be Run-Ready ### Docker Hub Just as you can store and share code using git version control on GitHub, you can also store and share images on [Docker Hub](https://hub.docker.com/). Docker Hub is a cloud-based service that lets you store images in private or public repositories (like GitHub!). This will come in handy when you want to deploy a Docker image onto a remote server (which you will set up later through Google Cloud) -- you'll create your image locally on your own laptop, push it to a repository on Docker Hub, go to your remote server that you've set up, "pull" the image from Docker Hub, and deploy it! ### Docker Compose We use Docker Compose in this course to help us get a glimpse of what a developer would use to manage an application that has multiple containers, each supporting a certain feature, acting as a single cluster. To give an example, our iOS app, Ithaca Transit, has multiple containers running, and here are a few examples of the instances that we need: * one to get data on where the TCAT buses are, and how delayed they are * one to compute bus routes * one to compute walking routes To manage each of these containers on their own respective servers can be quite tedious and frustrating for developers--which is why we take advantage of Docker Compose to help us centralize all of these different instances of running containers into one place, one server. Docker explains Docker Compose in [their docs](https://docs.docker.com/compose/) as: > a tool for defining and running **multi-container** Docker applications. With Compose, you use a **YAML** file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. Which is exactly what we need! #### #### What's a YAML file? The use of a "YAML" file (simply a text file with the file extension of `.yml`) is incredibly powerful, as it is what we outline to tell our remote servers later how to and what to deploy on our servers using Docker commands. Here's an example of what an example `docker-compose.yml` might look like: {% hint style="info" %} Deciphering a `docker-compose.yml` file: * The `build: .` on line 4 just tells Docker to build your application in the current directory. * You can have multiple services (which is the advantage of using Docker Compose!), and you can do so by specifying another chunk after `web: ... redis`with a different port number, like `3000` instead of `5000` for example. {% endhint %} ``` version: '2.0' services: web: build: . ports: - "5000:5000" volumes: - .:/code - logvolume01:/var/log links: - redis redis: image: redis volumes: logvolume01: {} ``` {% hint style="danger" %} This is not to be confused with a `Dockerfile`, which although is also a text document, it instead contains all the commands a user could call on the command line to assemble an image of an application. In short, it's a list of TODOs (ex. install python, install flask) needed in order to get *one* image of an application up and running anywhere--could be a local computer or a remote server. {% endhint %} {% hint style="success" %} **So what's the relationship between a `Dockerfile` and a `docker-compose.yml`?** You must have a `Dockerfile` for each image of an application, and you can specify multiple images in a `docker-compose.yml`, since it's an orchestration tool that we mentioned earlier for the purpose of clustering. This can seem confusing, but don't worry too much about it for now. For the scope of this course, we won't be deploying multiple images for an application. We will focus on making *one* `Dockerfile` to build *one* image, which we will put in our `docker-compose.yml` so that we can deploy *one* container on our server. But in the real world however, like in the case of Ithaca Transit, we would specify multiple `Dockerfile`s, each for a particular service, build their respective Docker images, and specify all of those images in one `docker-compose.yml`, so that we can deploy them as multiple containers at once. {% endhint %} ## Containerization Summary After you have isolated any **environment variables** out of your code, you can begin the two high level steps of containerization: 1. Compress code into a production environment 2. Prepare production environment to run-ready Stringing all of the steps in both together looks as follows: 1. Define a `Dockerfile` to specify your application's required environment 2. Build a **docker** **image** with your `Dockerfile` blueprint with the command: `docker build .` 3. Push your **image** to Docker Hub so that we can pull it onto our server later 4. Define how to run your image(s) in a `docker-compose.yml` file which can handle a multi-container application (in the scope of this course, you will only have one service) {% hint style="info" %} In the next chapter, we will continue the deployment process by pulling our docker image onto a remote server and running our `docker-compose.yml` file to start up our application! {% endhint %} # Demo Contributors: Alicia Wang, Conner Swenberg Suppose we have an application with a similar structure to our previous assignments: ```yaml src/ |-app.py |-db.py |-requirements.txt ``` How do we containerize this application? ## Setup image configuration A Dockerfile tells Docker what exactly we need to build into our image to run our application and exist in our `src` directory where the rest of the application code lives. {% code title="Dockerfile" %} ```yaml FROM python:3.7 RUN mkdir usr/app WORKDIR usr/app COPY . . RUN pip install -r requirements.txt CMD python app.py ``` {% endcode %} ### `FROM python:3.7` This `FROM` command will set up our environment from a prebuilt Python distribution that Docker provides. We will be using Python 3.7, but a list of other distribution versions can be found [here](https://hub.docker.com/_/python). ### `RUN mkdir usr/app` This `RUN` command executes the shell command `mkdir` to create the directory `usr/app` in the root directory ### `WORKDIR usr/app` The `WORKDIR` command sets the working directory for any subsequent `RUN`, `CMD`, `ENTRYPOINT`, `COPY` and `ADD` instructions that follow it in the `Dockerfile`. We will be using three of them. ### `COPY . .` This `COPY` command will copy all of our files in our current local directory `.` (which is `src` in our demo) to the root directory in the Docker image. ### `RUN pip install -r requirements.txt` This `RUN` command will execute the shell command `pip install -r requirements.txt`, which will install our required Python packages. ### `CMD python app.py` This `CMD` command will run our Python server when instantiating our container. ## Ignore unnecessary files There are some automatically generated files that we don’t need to run our application, so we want to tell Docker to ignore those files when copying our source code over. We can do this with a `.dockerignore` file: {% code title=".dockerignore" %} ```yaml __pycache__ venv ``` {% endcode %} Any files or directories listed in `.dockerignore` will not be included when copying our files with `COPY . .`. ## Build an image We can build an image using the Dockerfile located in the current directory: ``` >>> docker build . ``` To list all previously built images: ``` >>> docker images ``` We should see that the image was created with a specific image ID. These IDs are hard to keep track of, so we can give them a **tag** (or an alias) so that we know which image corresponds to which program. If we wanted to name our newly created image “demo”, we could run: ``` >>> docker build -t demo . ``` ## Run a container To run the application specified on the image: ``` >>> docker run -p "5000:5000" -it demo ``` The `-p` flag represents the **port mappings**. Because the application runs on a specified port but that port isn’t exposed outside of the container, we want to specify that the app’s port 5000 will listen to the container’s port 5000. The `-it` flag lets us run the container interactively, meaning that we can `ctrl-c` (force stop) the session if we so desire. #### docker-compose These flags can be hard to keep track of, so if we want to run the container a certain way every time, we can instead specify a docker-compose file which will specify any extra settings you need. In addition, you can use docker-compose to run multiple container applications as one. {% code title="docker-compose.yml" %} ```yaml version: "3" services: demo: image: demo ports: - "5000:5000" webapp: image: webapp ports: - "3000:3000" depends_on: - demo ``` {% endcode %} Now to run a container(s) from an image(s), instead of using `docker run`, we can instead use: ``` >>> docker-compose up -d ``` The `-d` flag is the detached mode, meaning that the container(s) will run in the background. \ To enter a running container and view and interact with it with `bash` commands: ``` >>> docker exec -it [CONTAINER_ID] bash ``` To stop running the current container: ``` >>> docker-compose down ``` ## Stop a running container To (forcefully) stop a running container, you will need its container ID, which you can find by viewing all running containers: ``` >>> docker ps ``` Then to stop the container: ``` >>> docker kill [CONTAINER_ID] ``` `>>> docker kill [container_id]` ## Docker Hub Docker Hub is an online service that allows us to open-source the images we built. We can share them with others who may want to run our application. To use Docker Hub, you must first create an account on Docker Hub's website, and then login to your account locally: ``` >>> docker login ``` DockerHub requires image tags to be unique. We can ensure our tags’ uniqueness by having our tags be prepended with our account username: ``` >>> docker tag demo cswenberg/demo ``` Finally we can push our built image to DockerHub, where it can be pulled by anyone else: ``` >>> docker push cswenberg/demo ``` # Assignment Handout Contributors: Alicia Wang, Conner Swenberg, Alanna Zhou, Shungo Najima ## Assignment Scope {% hint style="warning" %} **Due Date:** Monday 4/28, 11:59pm {% endhint %} You will be creating a Docker image of your CMS assignment and pushing it to Docker Hub! We will be grading your submissions by simply running `docker-compose up -d`, so make sure that your `docker-compose.yml` file is pointing to your Docker Hub image, not a local one! ### 1. Use your existing PA4 code Make sure that you use what you submitted for PA4, and that at least one endpoint works (you can note this in your README.txt). We will not be running test cases against your PA4 code to grade your PA5 submission. Then, create your `Dockerfile` and `docker-compose.yml` in your `src` folder as we did in the demo! ### 2. Create your Dockerfile & Docker Compose File, Create a Docker Image, then Push Image to Docker Hub Watch the demo if you forgot how to do this! Make sure you name your Docker image `/` . Kate did this in demo as well. Not doing this will lose you some style points! ### 3. Replace your current README.txt Replace your current README.txt with the one below. Fill out the required fields as usual, and answer the questions for full credit! If you have trouble answering the questions, watch the lecture video. Each question is answered there. {% file src="/files/FQnolVW0OpTNCjxxhR17" %} ### 4. Submit Assignment #### 4a. Fill out your README.txt You should have a newer version of README.txt as mentioned above! #### 4b. Zip and submit your assignment files Make sure that your `docker-compose.yml` file **is pointing to your Docker Hub image, not a local tag**. We will grade your submissions by running `docker-compose up -d` and then testing the route you designated in your README to see if your application is running properly as a Docker container. **You can check that your Docker Hub image exists if you go to** [**https://hub.docker.com/**](https://hub.docker.com/) **and find under your Docker Hub username, the tag that you've pushed!** Next, zip the starter folder and submit on CMS. For reference, your directory structure should look like the following: ``` pa5/ |-README.txt |-postman_collection.json |-src/ |-app.py |-db.py |-Dockerfile |-docker-compose.yml |-requirements.txt ``` For clarification, this means that you **SHOULD NOT** include your virtual environment, pycache, or .db file in your final submission. Doing so will lose you a few points on the project. #### 4c. Submit Feedback Form To receive credit for your assignment, you **must** fill out the feedback form (link on the left sidebar of the textbook) so that we can better understand how effectively we are teaching and how long students are spending on assignments. ## Challenges ### **Tier I** **+1 point** #### **Use a Different Python Distribution** Choose a different python distribution to use other than a basic version of python and write about what makes that distribution unique and its potential use cases. We expect upwards of 3 sentences in your README. ### **Tier II** **+1 points**\ **+1 point more if Tier I completed** #### **Add Gunicorn Workers** When running our application on a server (our end goal), there are many improvements we can make to our application with regards to how we run it. `python app.py` is a basic way to run an application, but we can try other methods. Gunicorn is one of these methods and you can read about it [here](https://vsupalov.com/what-is-gunicorn/). For this exercise, we want you to use Gunicorn to spawn 4 "workers" to run our application, essentially quadrupling our ability to handle incoming requests (hardware permitting). To install gunicorn, simply `pip install gunicorn` and don't forget to add your version to the `requirements.txt` file! Read [here](http://docs.gunicorn.org/en/stable/run.html#commands) about crafting your gunicorn command. Once you have your desired Gunicorn command to spawn our additional workers, we recommend writing it in a `start_server.sh` file. We can execute a `.sh` file with a `sh .sh` command. We like to place our Gunicorn command in this `.sh` file because it makes our lives easier by not having to remember the exact command every time we want to run our app. Now replace `sh start_server.sh` as our new command to run in our Docker container (don't forget to rebuild your image with the new requirements and `start_server.sh` file!). ### Total Points to Gain: 3 # 6. Deployment {% content-ref url="/pages/-Lu9g4XfsHZpl-4ttA6z" %} [Lecture](/chapters/deployment/lecture) {% endcontent-ref %} {% content-ref url="/pages/-MN0jnKkUhXua7OCpjfl" %} [Demo](/chapters/deployment/deployment-demo) {% endcontent-ref %} {% content-ref url="/pages/-MLQQOINWNv9vUv969hJ" %} [Assignment Handout](/chapters/deployment/assignment-handout) {% endcontent-ref %} # Lecture Contributors: Alicia Wang, Conner Swenberg ## Slides {% file src="/files/uXfC5JutGTbyROKptlZ7" %} ## How do other people use our application? As we learned in the first lecture, requests are made to servers. For our application to be networked, we too need to get our code running on a publicly accessible server. This process of publishing code into a real, running application is called **deployment**. Outline of deployment process: 1. Compress code into a production environment 2. Prepare production environment to run-ready 3. Spin up server(s) 4. Download & run prepared production environment on server(s) Last time we discussed **containerization**, which covers the first two steps in this process. Now we will continue and review the final two steps of actually running our application code so that it is a publicly accessible API. Let us first revisit servers. ## Servers As you may recall from the first chapter on Routes, a **server** is a software or hardware device that accepts and responds to requests made over a network (i.e. the internet, a local network, etc.), while a **client** is a device that makes the request, and receives a response from the server. ### Some Examples of Servers {% hint style="info" %} Don't worry about these details, they're just here as examples to get your head around on what a server could look like! {% endhint %} #### Web server * Web servers show pages and run apps through web browsers * The client is the web browser, like Chrome, Firefox, Safari #### Email server * Email servers manage sending and receiving emails * An email client can be a desktop application (Microsoft Outlook, iOS Mail) or simply web-based applications that access email on a server through a browser (Gmail page in the Mozilla Firefox browser) * The email client on your computer connects to an IMAP or POP server to download emails, and a SMTP server to send emails #### Identity server * Identity servers manage logins and security roles for authorized users * A good example is logging into Cornell's student center but making a pit stop at CUWebLogin with your NetID and password -- that process of authenticating a Cornell student is done through making a request on an identity server and getting a valid response back ### How a server connects to a network An internet server is assigned an **IP address** by the Internet Network Information Center (InterNIC, established by the US National Science Foundation) or by a web host (a company in charge of web pages, ex. GoDaddy). This IP address is what distinguishes the server from the others on a network when the server connects to a **router** or a switch. A router is a hardware device that receives, analyzes, and moves data to another network, and a common example is the Wireless (Wi-Fi) router. ### How users connect to a server Users connect to a server by using its domain name (ex. google.com), which is translated to the server's IP address by a DNS resolver. For example, the domain name "google.com" points to the address `216.58.216.164`, and the DNS (domain name system) server is what converts the domain name into the IP address on the internet. ### Where are servers stored? Servers can vary from your own computer or equipment in companies that are stored away in closets. Servers are often remote--meaning that they are located in data centers, managed by some company. Later, when you "create" your own server, you'll be getting it from Google Cloud, which is essentially a data center managed by Google. ## Deployment ### Running an Application: Our Needs In order to allow users to utilize our applications at all times, our server needs to always be running and open to receiving requests. The server should also be publicly accessible on the web. So far we've been running our servers locally, which is impractical if we need to scale up our application. Instead, we can use more powerful hardware that is managed by a third-party to run our server(s). Cloud services are able to run our containerized software exactly as we would locally. Popular services include Amazon Web Services, Google Cloud, and Microsoft Azure. These companies will handle the physical maintenance of the server and will provide tools to automate security, scaling, and crash handling. ### Accessing a Server To access our server remotely, we use Secure Shell (SSH) network protocol. This protocol gives users a secure way to access another computer, where users can open and view resources, execute commands, install packages, and update software. For security, users must provide credentials in the form of keys and passwords. SSH can either use automatically generated public-private key pairs to encrypt the connection and request password credentials for the user, or users can use a manually generated public-private key pair to log in without a password. In short, SSH ensures secure transfer of information between the host and the client. **Host** refers to the remote server you are trying to access, while the **client** is the computer you are using to access the host. ```bash >>> ssh user@server-ip >>> ssh user@domain.com ``` If you have never connected to this server before, it will ask you to confirm access, and your computer will add the IP to a list of `known_hosts` in your home directory so that it remembers and trusts the source. You will be validated as follows. Upon SSH request, will be prompted to provide the server's key and a request to connect. Your computer will provide the public/private key pair or your credentials. If successful, you will be granted access, logged in to the server and open a command prompt. ### Google Cloud's Web Terminal Instead of SSHing into our server from the command line, we can also use Google Cloud's web terminal to access the server. We are automatically authenticated by logging in with our Google account when we pull up the web command prompt interface. The functionality is exactly the same as if we were to SSH through your command line. The web terminal just simplifies the process for newcomers. ## Clustering ### Overview For any given application, a server may have to balance many different roles within the application. Typical role examples are dedicating a port to run a web application (what we have been doing), operating a relational database management system (RDBMS), or handling the storage of more freeform data like random files. In industry, service providers like Amazon Web Services (AWS) have different servers dedicated to each aspect of this system. You have EC2 instances for running application code, RDS instances for managing databases, and S3 buckets for storing the raw content of files (images, documents, etc.). ![Server operating multiple processes simultaneously for an application](/files/-LwYgwMqxWZ1evQgmhNw) The goal of clustering is to create a group of servers that connect in a way to behave as one system. By delineating the specific role of a server, cloud computing services like AWS and GCloud can optimize each server class to perform this task. When connected properly to allow for fluid communication, you can create a better performing system than one instance handling all the roles of running an application. ![Connecting various responsibilities across multiple servers](/files/-LwYh0b7d8EDkStxhc4F) ### Benefits The increased modularity of our system gives our application more resiliency and fine-tune control. We can backup versions of our database at different times, add specific server downtime-handling methods all to improve our data and application resiliency. By separating these roles, we also open up an easier way to manage these systems. Need a database capable of more operations/second or net storage size? Increase our database server specs. Want to add a new aspect of the application that demands storing a new file type? Add a new subdirectory within a bucket or create a new one entirely. Need to handle increased user traffic? Increase our web server specs. By separating our application's needs across multiple places, we gain a new level of orthonormality. ### Load Balancing Another cool extension of clustering is **load balancing**. The basic idea with load balancing is to have many servers running the same web app. For the exposed IP accepting requests, we have a server whose duty is just to reroute requests to one of the many servers available. In our example, we have one load balancer that has three web servers available to route too, therefore lowering the load on each server by a factor of three. ![Load balancer directing network traffic between multiple web servers](/files/-LwYh3vqqd9-tbip9uQO) In the case of a steep increase in user traffic, we can automate a process of spinning up new servers to give the load balancer more options to split the load, still allowing us to give speedy responses to clients. ![Increasing number of web servers available to load balancer](/files/-LwYhAS5RwfqyekrZW9N) # Demo Contributors: Alanna Zhou, Shungo Najima, Joshua Dirga Watch Demo Video Uploaded on the AppDev YouTube channel! ## Steps to deploy your project to a virtual machine in Google Cloud! 1. Prepare your project. Make sure it is ready to be containerized You may use this Dockerfile template here. {% code title="Dockerfile" %} ``` FROM python:3.7 RUN mkdir usr/app WORKDIR usr/app COPY . . RUN pip install -r requirements.txt CMD python app.py ``` {% endcode %} 2. Configure your billing, for students enrolled in our course, navigate to [Google Cloud](https://cloud.google.com/) and Start Free Trial or Redeem Credit. 3. Create your VPC Network 1. Name: name it anything you want! 2. Description: anything you want (optional) 3. Enable automatic subnet creation 4. Under Firewall Rules: enable Allow SSH 4. Create your VM 1. Machine Configuration (First tab on the left) 1. Name: name it anything you want! 2. Region: anyone should be fine, but closest to Ithaca is Northern Virginia. 3. Machine type: any, but to save costs we use e2-small 2. OS and storage (Second tab on the left) 1. Click "Change" and change the operating system to Ubuntu 3. Networking (Fourth tab on the left) 1. Under firewall, check "Allow HTTP traffic" 2. Under network interface, set the network to the VPC you created just now 4. Click "Create" to create you VM! 5. Build your Docker image locally and push to Dockerhub 1. `docker build --platform=linux/amd64 -t joshuadirga/demo-5:v1.0.0 .` \ Make sure to set the platform flag to be `linux/amd64` because our VM is using Ubuntu, a Linux based OS. Remember that the format for the Docker image is `accountname/reponame:tagname` 2. Run `docker push joshuadirga/demo-5:v1.0.0` to push to Dockerhub 3. Check that your image is on [Dockerhub](https://hub.docker.com). 6. SSH into your VM by clicking the "SSH" button under the "Connect" column 7. Follow steps 1 on the [Docker installation guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-22-04) and the [Docker Compose installation guide](https://www.digitalocean.com/community/tutorials/how-to-install-docker-compose-on-ubuntu-18-04) to install Docker and Docker Compose. (Make sure in the drop down choose the one with the latest ubuntu version) 8. Pull your Docker image from Dockerhub using\ `sudo docker pull joshuadirga/demo-5:v1.0.0`\ Check that it successfully pulled by listing the images on your VM using\ `sudo docker images` 9. Create a docker-compose.yml file using vim. `vim docker-compose.yml`\ This will create a new docker-compose.yml file if it has not been created already.\ \ **Some useful VIM commands:**\ \- press "i" on the keyboard to enter insertion mode (you can only edit in insertion mode)\ \- press "esc" to escape insertion mode\ \- once escaped from insertion mode type `:w` to save file or `:q` to exit VIM. Use `:wq` to save and exit at the same time.\ \ Make sure the image name is the same one you pushed to Dockerhub earlier. Remember that the port `80:5000` refers to `container's port : app's port`. The container's port is the port you are running you Docker Container on inside the VM. The app's port is the port inside the container, where you are running your app (remember you app.py file uses the port 5000). Here is a template you may use for the docker-compose.yml! {% code title="docker-compose.yml" %} ``` version: '3' services: demo: image: joshuadirga/demo-5:v1.0.0 ports: - "80:5000" ``` {% endcode %} 10. Check that you docker-compose.yml exists by running `cat docker-compose.yml` 11. Run you app using `sudo docker-compose up -d` to run in detached mode 12. Check that its running using `sudo docker ps` to list all the running containers. Copy your container's id, and then use the command \ `sudo docker logs YOURCONTAINERID`\ To view the logs from your container. The logs should be similar to what you see when you run your app locally! 13. On GCP (Google Cloud Platform), click on the External IP for your VM to access the backend on your browser. Copy the URL over to postman to test, and CONGRATULATIONS YOU'VE MADE YOUR FIRST DEPLOYMENT!! 14. Run `sudo docker-compose down` to kill your container. The next time you run, just run `sudo docker-compose up -d` again because your images are still in your VM! If you made a change to your backend code and want to redeploy, here are the steps: 1. Build using `docker build --platform=linux/amd64 -t joshuadirga/demo-5:v1.0.0 .` (Do this locally on your laptop's command line) 2. Push using `docker push joshuadirga/demo-5:v1.0.0 .` (Do this locally on your laptop's command line) 3. Pull from Dockerhub using `sudo docker pull joshuadirga/demo-5:v1.0.0` (Do this in your server) 4. Use `sudo docker-compose up -d` to run! Additional Tips: * You only need to change your docker-compose.yml if you are changing any of the configurations in running your container * A lot fo the problems encountered in this deployment setup is port issues, so remember to use port `80:5000` or something like `8080:5000` in your docker-compose.yml file. # Assignment Handout Contributors: Alicia Wang, Alanna Zhou, Shungo Najima ## Assignment Scope {% hint style="warning" %} **Due Date:** Monday 5/5, 11:59pm {% endhint %} You will be deploying from the `Dockerfile` you created in PA5 to Google Cloud! It should have at least one functioning endpoint from PA4. Google Cloud is a platform that allows developers to quickly build and deploy apps in the cloud. We hope that this assignment will help you get familiar with Google Cloud so that you can use it for your own remote servers during the hack challenge and in your personal projects. ### Deploy Your Server Here are the requirements for your server: * The server has to be publicly accessible (the course staff must be able to access its URL) * You must have a working endpoint defined in the `README.txt` * You must have an environment variable file on your server containing your netid and your server must have a base endpoint as shown below. This means making changes to your `app.py`. \ Note that to receive full credit, your base endpoint must read from your environment variable file (e.g. you cannot simply hardcode the string "sn685 was here!") **`GET`**`/` {% code title="Success Response" %} ```javascript " was here!" ``` {% endcode %} ### 3. Replace your current README.txt Once again, replace your current `README.txt` with the one below. If you'd like to save your PA 5 answers, you may want to move the `README.txt` from that assignment into a different folder. Fill out the required fields as usual, and answer the questions for full credit! If you have trouble answering the questions, watch the lecture video. Each question is answered there. {% file src="/files/6lHbYKwJ514Cgiu8YCle" %} ### 4. Submit Assignment #### 4a. Fill out your README.txt You should have a newer version of README.txt as mentioned above! #### 4b. Zip and submit your assignment files If you have properly followed along with the demo, you should now newly have a `.dockerignore` file, a `.env` file, and a few changes to your `app.py` and `docker-compose.yml`. Zip the starter folder and submit on CMS. For reference, your directory structure should look like the following: ``` pa6/ |-README.txt |-postman_collection.json |-src/ |-app.py |-db.py |-Dockerfile |-docker-compose.yml |-.dockerignore |-requirements.txt |-.env ``` For clarification, this means that you **SHOULD NOT** include your virtual environment, pycache, or .db file in your final submission. Doing so will lose you a few points on the project. #### 4c. Submit Feedback Form To receive credit for your assignment, you **must** fill out the feedback form (link on the left sidebar of the textbook) so that we can better understand how effectively we are teaching and how long students are spending on assignments. ## Challenges None for this assignment! Feel free to explore what Google Cloud has to offer, which is a lot. # 7. Images {% content-ref url="/pages/-MN0jcKBPfI0UGYwAEKG" %} [Demo](/chapters/7.-images/demo) {% endcontent-ref %} {% content-ref url="/pages/AaGdMwgHSnZmyLl4Eolq" %} [Assignment Handout](/chapters/7.-images/assignment-handout) {% endcontent-ref %} # Demo Contributors: Alanna Zhou, Shungo Najima ## 1. Download Starter Code {% file src="/files/R4AGB1NKTMSLua5OBEmR" %} ## 2. Watch Demo Uploaded on the AppDev YouTube channel! # Assignment Handout Contributors: Shungo Najima #### Submit Feedback Form Your only TODO for this week is to fill out the feedback form (link on the left sidebar of the textbook) so that we can better understand how effectively we are teaching. Please put the assignment difficulty as 3 and the hours you spent as N/A for this week. # 8. Authentication {% content-ref url="/pages/-Lu9gp7yHcnKHM2kSfyO" %} [Lecture](/chapters/authentication/lecture) {% endcontent-ref %} {% content-ref url="/pages/-Lu9gstXK5uCgPvgUqFM" %} [Demo](/chapters/authentication/demo) {% endcontent-ref %} {% content-ref url="/pages/yZZuO2jT2SoYi9VvwObI" %} [Assignment Handout](/chapters/authentication/assignment-handout) {% endcontent-ref %} # Lecture Contributors: Alicia Wang, Conner Swenberg In any sort of application involving users, we need a method for verifying identities. ### Overview When making requests to a server, how does the server know who you are and how does the server validate? Authentication is a process by which we ensure and confirm a user's identity. Example: Upon signing up for an account for some service, they will store your username/email and password credentials (hashed). To later access your account, you login with the same credentials and they check if that same combination exists ![Adding new credentials to database upon sign-up](/files/-LwYiMQ7rFR5G2QUt-JN) ![Checking against existing credentials upon log-in](/files/-LwYibgVprAk3OCmyBH9) ## HTTP Basic Authentication 1. Client sends your username and password to server 2. Browser caches (saves) your credentials temporarily 3. Client sends credentials with every request in the header to let the server know your identity An example of a request from the client with authentication headers: ``` GET /myfiles/ HTTP/1.1 Host: www.google.com Authorization: Basic Zm9vOmJhcg== ``` `Zm9vOmJhcg` represents a base64 encoding of your username and password and can be decoded to extract your credentials. Basic authentication has problematic security risks. Your username and password credentials are exposed in every request (even when encoded), so anyone looking at your network traffic would have access to that data. In addition, your credentials are stored until the browser is closed or you clear your cache, so there is no easy way to "log out". ## HTTP Session Authentication Session-based authentication sets up a time period for when a client and server can freely communicate. Sessions are stored as a separate table in the database and are unique to each user. A session object will have (at minimum) a foreign key referencing a user and an expiration datetime. ### Authentication Process 1. Client signs in (authenticates) with its credentials 2. Server validates the credentials 3. Server creates and returns a session token for the user 4. Client stores this token as a cookie in the browser or on disk 5. Client sends the token with every request so the server can identify the user ### Token Expiration We design for tokens to expire after a certain length of time. We do this so that any accidental exposure of the token does not have any lasting effects and give someone else permanent permissions. More advanced session authentication schemes involve the ability for a user to easily refresh their session with a second token, called a **refresh token**. This refresh token is designed to only be used to retrieve a new session token and lasts for a lot longer (potentially on a scale upwards of weeks). ### Table Design ```sql CREATE TABLE session ( id INTEGER PRIMARY KEY, sessionToken TEXT NOT NULL, expiresAt INTEGER NOT NULL, refreshToken TEXT NOT NULL, userID INTEGER NOT NULL, FOREIGN KEY (userID) REFERENCES user(ID) ) ``` Find a user ID associated with a session: ```sql SELECT userId FROM session WHERE sessionToken = ``` Find a user associated with the user ID: ```sql SELECT * FROM users WHERE id = ``` ### Benefits Requests only contain a session token to prove identity, thus minimizing the exposure of usernames and passwords over requests. By designing these session tokens to expire after a short period of time, we limit potential for accidental access to be granted to unauthorized individuals. # Demo Contributors: Conner Swenberg, Alanna Zhou, Shungo Najima ## Download Demo Starter Code {% file src="/files/lfvHhwDLsHZ28WXdyNNE" %} ## Watch Demo Video Uploaded on the AppDev YouTube channel! # Assignment Handout Contributors: Shungo Najima #### Submit Feedback Form Your only TODO for this week is to fill out the feedback form (link on the left sidebar of the textbook) so that we can better understand how effectively we are teaching. Please put the assignment difficulty as 3 and the hours you spent as N/A for this week. # (Work in Progress) OAuth {% content-ref url="/pages/-LxUchwQ-10NmL8x9QbI" %} [Lecture](/chapters/oauth/lecture) {% endcontent-ref %} {% content-ref url="/pages/-LxUckSR3Y1PSmUo8izi" %} [Demo](/chapters/oauth/demo) {% endcontent-ref %} # Pre-Class TODO's ### 1. Watch Pre-Lecture Video We will not be conducting a typical keynote lecture in-class, just a demo for the full hour. {% hint style="info" %} Coming Soon! {% endhint %} ### 2. Download Demo Starter Code {% hint style="info" %} Coming Soon! {% endhint %} # Lecture Contributors: Conner Swenberg ## OAuth 2.0 You have most likely interacted with OAuth already via "Login with Facebook" or "Login with Google". Either of these options allows you to authenticate yourself without having to use a separate username and password for a particular service, just piggy-back off of another service's authentication. | Stakeholder | Benefit | | --------------------------------------- | ------------------------------------------------- | | Service / application | Streamlines user registration / login | | Users "resource owners" | Minimizes number of accounts | | API "resource server" (Facebook/Google) | Gains valuable data on your use of other services | ### Registration Our service that intends to use OAuth needs to register itself with the resource server, receiving their own **client ID** and **client secret.** ### Redirect URI Our service also needs to declare to the resource server a **redirect URI**. This tells the resource server where to bring users back to after logging with their authentication system. ### Code Generation 1. Service pulls up a custom resource server login page unique to their client ID and redirect URI. 2. Client returned a new code upon successful login 3. Client will use this code to fetch a session token from the resource server ## BELOW IS UNDER DEVELOPMENT ## OAuth 1.0 [creds](https://docs.authlib.org/en/latest/basic/oauth1.html) ### Roles ### Credentials ### Flow ### Signature ## OAuth 2.0 [creds](https://docs.authlib.org/en/latest/basic/oauth2.html) ### Roles ### Flow ### Grant Types #### Authorization Code #### Implicit #### Resource Owner Password #### Client Credential ### Client Authentication Models ### Scope and Access Control ## OpenID Connect ### Claims # Demo Contributors: Conner Swenberg ## Create a Google Client ## Packages ### Flask-Login ### OAuthLib ## User Model ## Authentication Routes ### Setup 1. Login: `login/` 2. Login Callback: `callback/` 3. Logout: `logout/` ### Login ### Login Callback ### Logout ### Requiring Authentication on Normal Routes # HackOurCampus Contributors: Alanna Zhou ## Pre-Workshop TODOs Welcome to AppDev's Backend [HackOurCampus](https://hackourcampus.com/) workshop! Here are some things to complete before coming so that you're prepared (ﾉ◕ヮ◕)ﾉ\*:･ﾟ✧ The Fall 2020 course instructors, Alanna Zhou and Alicia Wang, will be leading this workshop--don't be afraid to find us on the hackathon [Discord]() to reach out and ask us questions! ### 1. Install Postman {% content-ref url="/pages/-Lw\_IsDUKu2sa6KDel87" %} [Postman](/cheat-sheets/postman) {% endcontent-ref %} ### 2. Familiarize Yourself With Command Line Basics {% content-ref url="/pages/-Lw\_J5gEPWGYjX\_aBA5n" %} [Command Line](/cheat-sheets/command-line) {% endcontent-ref %} ### 3. Install a Text Editor Any text editor like Vim, Sublime Text, Atom, or Emacs all work just fine! Our course instructors and TAs prefer VSCode if you can't choose. {% embed url="" %} ### 4. Install Python Before you do anything, make sure that you have a version of Python3 installed (Python 3.7.6 is a good choice if you can't decide). Python2 may work, but you might run into some package issues. {% embed url="" %} Check your Python versions to make sure you'll be using the right one. Notice that if you have both Python2 and Python3, you may have to use `python3` when you're running it as a command so that your computer knows only to use version 3. ``` $ python --version Python 2.7.15 $ python -V Python 2.7.15 $ python3 --version Python 3.7.6 $ python3 -V Python 3.7.6 ``` ### 5. Download Starter Code Make sure to unzip this and that you're able to open this in your text editor once the workshop starts! {% file src="/files/-MFRpWrlsNQfwuB\_K\_6N" %} Starter Code {% endfile %} ### 6. Install Packages Once you unzip the starter code, you should have a folder named `src` with three items: * `app.py` * `db.py` * `requirements.txt` We will be writing code in `app.py` and `db.py`. However, in order to write that code, we'll have to install some packages, which are included in `requirements.txt`. You can install them by running `pip3 install -r requirements.txt`. You should already have `pip` on your computer, but here are some commands to help if you don't: #### MacOS & Linux ```bash pip install -U pip ``` #### Windows ```bash python -m pip install -U pip ``` # Previous Semester Syllabi # FA22 Syllabus Jessica Sylvester, Shungo Najima ### Course Staff
NameRoleOffice HoursLocation
Jessica SylvesterInstructorFriday 5-6 pmHollister 312
Shungo NajimaInstructorWednesday 4-5 pmUpson 216
Gonzalo GonzalezTAMonday 6:15-7:15 pmHollister 362
Sasha LoayzaTAMonday 6:15-7:15 pmHollister 362
Noah SolomonTATuesday 5-6 pmHollister 372
Bella HoieTATuesday 5-6 pmHollister 372
Marya KimTAWednesday 4-5 pmUpson 216
Kate LiangTAThursday 5-6 pmHollister 312
Joyce WuTAFriday 5-6 pmHollister 312
Kidus ZegeyeTASaturday 3-4 pmHollister 312
Mateo WeinerTASaturday 3-4 pmHollister 312
### Schedule {% hint style="info" %} This schedule is tentative and subject to changes to Cornell's Fall 2022 academic calendar. {% endhint %} Lectures are every **Monday** **and Wednesday** from 7**:30 pm to 8:20 pm** in **Olin Hall 255** and each week's lectures will be split into two parts: a concept-based overview of the week's topic, followed by a live implementation demo of the same topic. Additionally, there will be a project assignment every week, which will typically be due on the **following** **Wednesday**. You can find the lecture recordings on our [YouTube channel](https://www.youtube.com/channel/UCUDPjOx1yPRGxsO6AFtBZSg), for you to rewatch as you wish. | | | | | | ------------- | --------------------------------------- | ------------------------- | ---------------- | | Lecture Dates | Topic | Project | Project Due Date | | 2/21, 2/23 | Routes | Reddit | 3/5 | | 2/28 | February Break :) | N/A | N/A | | 3/2 | (Optional) Git | N/A | N/A | | 3/7, 3/9 | Databases | Venmo (Basic) | 3/16 | | 3/14, 3/16 | Relational Databases | Venmo (Full) | 3/23 | | 3/21, 3/23 | Abstractions | CMS | 3/30 | | 3/28, 3/30 | Containerization | Dockerize CMS | 4/6 | | 4/4, 4/6 | Spring Break!! | N/A | N/A | | 4/11, 4/13 | Deployment | Deploy CMS | 4/20 | | 4/18, 4/20 | Hack Challenge Kickoff + Authentication | Hack Challenge Submission | TBD | | 4/25, 4/27 | Images | Hack Challenge Submission | TBD | | 5/2, 5/4 | TBD | Hack Challenge Submission | TBD | | 5/9 | Hack Challenge Finale | N/A | N/A | ### Enrollment You can enroll yourself into the course like any other on Student Center upon passing the pre-test and submitting the application (see [Apply to Take the Course](/apply-to-take-the-course)). This is a 2-credit S/U course, however, we permit students to enroll for 1-credit for the purpose of avoiding a credit overload petition. All students are expected to do the same work, regardless of the number of enrolled credits. We highly encourage you to reference demos, attend office hours, and ask questions on Ed discussion. Along with submitting assignment source code, you will be required to submit a text file (README.txt) where you can freely comment on the week’s lecture, assignment, or anything else. You will also fill out a short questionnaire which helps us quantify feedback for improving the course. ### Grading As mentioned earlier, this is a **2 credit S/U course** and grading will be calculated as shown in the table below. Submission Deadline: **Wednesday 11:59 pm**\ Late Deadline: **Saturday 11:59 pm**\ Late Penalty: **-1 point penalty per day. We will not accept homework after the late deadline.**\ Safety Net: **6 total slip days for the semester**, **up to 3 can be used per assignment.** | | | | ------------------- | ------- | | Item | Weight | | PA1 - Reddit | 10 | | PA2 - Venmo (Basic) | 10 | | PA3 - Venmo (Full) | 10 | | PA4 - CMS | 10 | | PA5 - Dockerize CMS | 10 | | PA6 - Deploy CMS | 10 | | Final Project | 30 | | Weekly Surveys | 5 | | Attendance | 5 | | **Total** | **100** | | **Passing Score** | **70** | ### Hack Challenge All AppDev courses finish with bringing members from iOS, backend, and design together to put what you’ve learned to the test and build your very own application. All students are welcome to submit app ideas and teams up to four will be formed to bring the idea to life. Leading up to the Hack Challenge, the backend course will focus on deployment so that your mobile apps can connect to a live backend running on Google Cloud servers. It goes without saying that the majority of students find the Hack Challenge the most rewarding and fun part of the course! ### Academic Integrity As with any other course at Cornell, the code of academic integrity will be enforced in this class. While we encourage you to collaborate with other students, all code that you submit must be written by you (and although a great method for learning, pair programming does not abide by our standards of students submitting their own work). We also encourage using the internet to learn more about backend development, but again, any code you submit must be written by you. # SP22 Syllabus Kate Liang, Shungo Najima ### Course Staff
NameRoleOffice HoursLocation
Kate LiangInstructorTuesday 5-6 pmHollister 312
Shungo NajimaInstructorSaturday 4-5 pm,
Wednesday 6:30-7:30pm		Hollister 312
Noah SolomonTAMonday 5-6 pmHollister 372
Joyce WuTAMonday 5-6 pmHollister 372
Jessica SylvesterTATuesday 5-6 pmHollister 312
Mateo WeinerTAWednesday 6:30-7:30pmHollister 312
Gonzalo GonzalezTASaturday 3-4 pmHollister 312
Kidus ZegeyeTASaturday 3-4 pmHollister 312
Marya KimTASaturday 3-4 pmHollister 312
Raahi MenonTASaturday 4-5 pmHollister 312
### Schedule {% hint style="info" %} This schedule is tentative and subject to changes to Cornell's Spring 2022 academic calendar. {% endhint %} Lectures are every **Monday** **and Wednesday** from **8:35 pm to 9:25 pm** in **Phillips Hall 101** and each week's lectures will be split into two parts: a concept-based overview of the week's topic, followed by a live implementation demo of the same topic. Additionally, there will be a project assignment every week, which will typically be due on the **following** **Wednesday**. You can find the lecture recordings on our [YouTube channel](https://www.youtube.com/channel/UCUDPjOx1yPRGxsO6AFtBZSg), for you to rewatch as you wish. | | | | | | ------------- | --------------------------------------- | ------------------------- | ---------------- | | Lecture Dates | Topic | Project | Project Due Date | | 2/21, 2/23 | Routes | Reddit | 3/5 | | 2/28 | February Break :) | N/A | N/A | | 3/2 | (Optional) Git | N/A | N/A | | 3/7, 3/9 | Databases | Venmo (Basic) | 3/16 | | 3/14, 3/16 | Relational Databases | Venmo (Full) | 3/23 | | 3/21, 3/23 | Abstractions | CMS | 3/30 | | 3/28, 3/30 | Containerization | Dockerize CMS | 4/6 | | 4/4, 4/6 | Spring Break!! | N/A | N/A | | 4/11, 4/13 | Deployment | Deploy CMS | 4/20 | | 4/18, 4/20 | Hack Challenge Kickoff + Authentication | Hack Challenge Submission | TBD | | 4/25, 4/27 | Images | Hack Challenge Submission | TBD | | 5/2, 5/4 | TBD | Hack Challenge Submission | TBD | | 5/9 | Hack Challenge Finale | N/A | N/A | ### Enrollment You can enroll yourself into the course like any other on Student Center upon passing the pre-test and submitting the application (see [Apply to Take the Course](/apply-to-take-the-course)). This is a 2-credit S/U course, however, we permit students to enroll for 1-credit for the purpose of avoiding a credit overload petition. All students are expected to do the same work, regardless of the number of enrolled credits. We highly encourage you to reference demos, attend office hours, and ask questions on Ed discussion. Along with submitting assignment source code, you will be required to submit a text file (README.txt) where you can freely comment on the week’s lecture, assignment, or anything else. You will also fill out a short questionnaire which helps us quantify feedback for improving the course. ### Grading As mentioned earlier, this is a **2 credit S/U course** and grading will be calculated as shown in the table below. Submission Deadline: **Wednesday 11:59 pm**\ Late Deadline: **Saturday 11:59 pm**\ Late Penalty: **-1 point penalty per day. We will not accept homework after the late deadline.**\ Safety Net: **6 total slip days for the semester**, **up to 3 can be used per assignment.** | | | | ------------------- | ------- | | Item | Weight | | PA1 - Reddit | 10 | | PA2 - Venmo (Basic) | 10 | | PA3 - Venmo (Full) | 10 | | PA4 - CMS | 10 | | PA5 - Dockerize CMS | 10 | | PA6 - Deploy CMS | 10 | | Final Project | 30 | | Weekly Surveys | 5 | | Attendance | 5 | | **Total** | **100** | | **Passing Score** | **70** | ### Hack Challenge All AppDev courses finish with bringing members from iOS, backend, and design together to put what you’ve learned to the test and build your very own application. All students are welcome to submit app ideas and teams up to four will be formed to bring the idea to life. Leading up to the Hack Challenge, the backend course will focus on deployment so that your mobile apps can connect to a live backend running on Google Cloud servers. It goes without saying that the majority of students find the Hack Challenge the most rewarding and fun part of the course! ### Academic Integrity As with any other course at Cornell, the code of academic integrity will be enforced in this class. While we encourage you to collaborate with other students, all code that you submit must be written by you (and although a great method for learning, pair programming does not abide by our standards of students submitting their own work). We also encourage using the internet to learn more about backend development, but again, any code you submit must be written by you. # FA21 Syllabus Raahi Menon, Gonzalo Gonzalez ### **Course Staff** | Name | Role | | ---------------- | ---------- | | Gonzalo Gonzalez | Instructor | | Raahi Menon | Instructor | | Alicia Wang | TA | | Orko Sinha | TA | | Shungo Najima | TA | | Kate Liang | TA | Office hours will be announced soon! ### Schedule {% hint style="warning" %} Schedule subject to change throughout the Fall semester. {% endhint %} Lectures will be held in Gates G01 every Wednesday from 8:35 p.m. for 50 minutes. Lectures will be posted to the [Cornell AppDev Youtube channel](https://www.youtube.com/c/CornellAppDev) within 24 hours of the lecture in the Backend Course Fall 2021 playlist. Assigned homework will be due the following Tuesday at midnight. | Lecture Date | Topic | Project Topic | Project Due Date | | ------------ | ------------------------------- | ----------------------- | ---------------- | | 9/22 | Routes | Reddit | 9/28 | | 9/29 | Databases | Venmo (Basic) | 10/5 | | 10/6 | Fall Break Upcoming! | No Class! | N/A | | 10/13 | Relational Databases | Venmo (Full) | 10/19 | | 10/20 | Abstractions | CMS | 10/26 | | 10/27 | Containerization | Dockerize CMS | 11/2 | | 11/3 | Deployment | Deploy CMS | 11/9 | | 11/10 | Authorization | N/A | N/A | | 11/17 | Images/Frontend Integration | N/A | N/A | | TBD | Hack Challenge Opens (No Class) | Hack Challenge Midpoint | TBD | | TBD | Hack Challenge Cont. (No Class) | Hack Challenge Final | TBD | | TBD | Hack Challenge Finale | N/A | N/A | ### Enrollment You can enroll yourself into the course like any other on Student Center after passing the pretest and submitting the Google form (see [Apply to Take the Course](https://backend-course.cornellappdev.com/apply-to-take-the-course)). This is a 2-credit S/U course, however we permit students to enroll for 1-credit for the purpose of avoiding a credit overload petition. All students are expected to do the same work, regardless of the enrolled credits. ### Lectures Our lectures are broken down into two portions: a concept-based overview of the week's topic, and an implementation-based coding demo. Lectures and demos run for 10 weeks and will be held in Gates G01 Wednesdays at 8:25 pm. They will also be posted as videos on the Cornell AppDev [Youtube channel](https://www.youtube.com/channel/UCUDPjOx1yPRGxsO6AFtBZSg) within a day. The textbook chapters also serve as a great resource alongside the lectures. If you're ever confused, we highly recommend checking the associated textbook chapter or going to office hours - we're here to help! ### Weekly Assignments Programming assignments are due on Tuesdays at 11:59pm and submitted on CMS. **Late assignments will receive a flat -2 point penalty and no late submissions will be accepted after the Friday following the normal submission deadline. We only grant extensions to students who enroll in the course after the first lecture date.** Those who are granted extensions *must* make a private CampusWire post ***and*** upload their assignments to CMS so that instructors can grade and respond in a timely manner. Each assignment will have three optional challenges to take your understanding further. * Tier I challenges are designed to be low difficulty, simple extensions of the application (estimated time: <1 hour). * Tier II challenges are medium difficulty extensions that require more forethought for a successful implementation, but are by no means out of reach (estimated time: <2 hours). * Tier III challenges are high difficulty extensions which will often require outside research and will teach you things that we do not have time to cover in this course. \ We highly encourage that you reference demos, attend office hours, and ask questions on Campuswire. Along with submitting assignment source code, you will be **required** to submit a text file (`README.txt`) where you can freely comment on the week’s lecture, assignment, or anything else. You will also fill out a short questionnaire which helps us quantify feedback for improving the course. ### Grading This course is graded as S/U (pass/fail). We want all of our students to pass the course. As long as you turn in every assignment with reasonable effort and follow academic integrity, you should be on pace to pass. All assignments take into account effort with a minimum score if you submit something that shows you tried. No submission or noticeable lack of effort will receive zeros. **To pass this course, you will need >= 70** **points AND to create your own app by participating in the Hack Challenge.** We have many extra credit opportunities throughout the semester on assignments (up to 6 points each) as well as user research for apps that Cornell AppDev members are working on. Also note that although not mandatory, lecture attendance is highly recommended. | Assignment | Weight | | ------------------- | ------- | | PA1 - Reddit | 10 | | PA2 - Venmo (Basic) | 10 | | PA3 - Venmo (Full) | 10 | | PA4 - CMS | 10 | | PA5 - Dockerize CMS | 10 | | PA6 - Deploy CMS | 10 | | Final Project | 30 | | Weekly Surveys | 10 | | **Total** | **100** | | **Passing Score** | **70** | ### Hack Challenge All AppDev courses finish with bringing members from iOS, backend, and design together to put what you’ve learned to the test and build your very own application. All students are welcome to submit app ideas and teams up to four will be formed to bring the idea to life. Leading up to the Hack Challenge, the backend course will focus on deployment so that your mobile apps can connect to a live backend running on Google Cloud servers. It goes without saying that the majority of students find the Hack Challenge the most rewarding and fun part of the course! ### Academic Integrity As with any other course at Cornell, the code of academic integrity will be enforced in this class. While we encourage you to collaborate with other students, all code that you submit must be written by you (and although a great method for learning, pair programming does not abide by our standards of students submitting their own work). We also encourage using the internet to learn more about backend development, but again, any code you submit must be written by you. # SP21 Syllabus Raahi Menon, Jack Greenberg ### **Course Staff** | Name | Role | Office Hours | | -------------- | ---------- | --------------------------- | | Jack Greenberg | Instructor | Friday 3:00 pm - 4:00 pm | | Raahi Menon | Instructor | Sunday 6:00 pm - 7:00 pm | | Alanna Zhou | TA | Sunday 2:00 pm - 3:00 pm | | Orko Sinha | TA | Saturday 5:00 pm - 6:00 pm | | Tedi Mitiku | TA | Thursday 12:00 pm - 1:00 pm | | Shungo Najima | TA | Saturday 5:00 pm - 6:00 pm | | Noah Solomon | TA | Sunday 2:00 pm - 3:00 pm | | Kate Liang | TA | Sunday 6:00 pm - 7:00 pm | Office hours will be held at [this zoom link](https://cornell.zoom.us/j/98940362697?pwd=ZVduQVd6L0NDTms1YnNDcTR0WHBIUT09). Updates to office hours may be posted in the Slack, so keep your eyes peeled! ### Schedule {% hint style="warning" %} Schedule subject to change throughout the Spring semester. {% endhint %} Lectures will be posted to the [Cornell AppDev Youtube channel ](https://www.youtube.com/playlist?list=PLjf6nsEcF5KMhk0ZmTQzzWtvoEHY84B8p)asynchronously under the Backend Course Spring 2021 playlist every Monday at 7:30 p.m. EST. Assigned homeworks will be due the following Sunday at midnight (11:59 p.m. EST). | Lecture Date | Topic | Project Topic | Project Due Date | | ------------ | ------------------------------- | ----------------------- | ---------------- | | 3/1 | Routes | Reddit | 3/7 | | 3/8 | Databases | Venmo (Basic) | 3/14 | | 3/15 | Relational Databases | Venmo (Full) | 3/21 | | 3/22 | Abstractions | CMS | 3/28 | | 3/29 | Containerization | Dockerize CMS | 4/4 | | 4/5 | Deployment | Deploy CMS | 4/11 | | 4/12 | Authorization | N/A | N/A | | 4/19 | Images/Frontend Integration | N/A | N/A | | 4/26 | Wellness Day, No Class! | N/A | N/A | | 4/27 | Hack Challenge Opens (No Class) | Hack Challenge Midpoint | 5/2 | | 5/3 | Hack Challenge Cont. (No Class) | Hack Challenge Final | 5/9 | | 5/10 | Hack Challenge Finale | N/A | N/A | ### Enrollment You can enroll yourself into the course like any other on Student Center after receiving notification that you have passed the pretest (see [Apply to Take the Course](https://backend-course.cornellappdev.com/apply-to-take-the-course)). This is a 2-credit S/U course, however we permit students to enroll for 1-credit for the purpose of avoiding a credit overload petition. All students are expected to do the same work, regardless of the credits enrolled in. ### Lectures Our lectures are broken down into two portions: a concept-based overview of the week's topic, and an implementation-based coding demo. Lectures and demos run for 10 weeks and will be posted as videos on the Cornell AppDev [Youtube channel](https://www.youtube.com/channel/UCUDPjOx1yPRGxsO6AFtBZSg). The textbook chapters also serve as a great resource alongside the lectures. If you're ever confused, we highly recommend checking the associated textbook chapter or going to office hours - we're here to help! ### Weekly Assignments Programming assignments are due on Sundays at 11:59pm and submitted on CMS (not Piazza!). **Late assignments will receive a flat -2 point penalty and no late submissions will be accepted after the Wednesday following the normal submission deadline. We only grant extensions to students who enroll in the course after the first lecture date.** Those who are granted extensions *must* make a private Piazza post ***and*** upload their assignments to CMS so that instructors can grade and respond in a timely manner. Each assignment will have three optional challenges to take your understanding further. * Tier I challenges are designed to be low difficulty, simple extensions of the application (estimated time: <1 hour). * Tier II challenges are medium difficulty extensions that require more forethought for a successful implementation, but are by no means out of reach (estimated time: <2 hours). * Tier III challenges are high difficulty extensions which will often require outside research and will teach you things that we do not have time to cover in this course. \ We highly encourage that you reference demos, attend office hours, and ask questions on Piazza. Along with submitting assignment source code, you will be **required** to submit a text file (`README.txt`) where you can freely comment on the week’s lecture, assignment, or anything else. You will also fill out a short questionnaire which helps us quantify feedback for improving the course. ### Grading This course is graded as S/U (pass/fail). We want all of our students to pass the course. As long as you turn in every assignment with reasonable effort and follow academic integrity, you should be on pace to pass. All assignments take into account effort with a minimum score if you submit something that shows you tried. No submission or noticeable lack of effort will receive zeros. **To pass this course, you will need >= 70** **points AND to create your own app by participating in the Hack Challenge.** We have many extra credit opportunities throughout the semester on assignments (up to 6 points each) as well as user research for apps that Cornell AppDev members are working on. Also note that although not mandatory, lecture attendance is highly recommended. | Assignment | Weight | | ------------------- | ------- | | PA1 - Reddit | 10 | | PA2 - Venmo (Basic) | 10 | | PA3 - Venmo (Full) | 10 | | PA4 - CMS | 10 | | PA5 - Dockerize CMS | 10 | | PA6 - Deploy CMS | 10 | | Final Project | 30 | | Weekly Surveys | 10 | | **Total** | **100** | | **Passing Score** | **70** | ### Hack Challenge All AppDev courses finish with bringing members from iOS, backend, and design together to put what you’ve learned to the test and build your very own application. All students are welcome to submit app ideas and teams up to four will be formed to bring the idea to life. Leading up to the Hack Challenge, the backend course will focus on deployment so that your mobile apps can connect to a live backend running on Google Cloud servers. It goes without saying that the majority of students find the Hack Challenge the most rewarding and fun part of the course! ### Academic Integrity As with any other course at Cornell, the code of academic integrity will be enforced in this class. While we encourage you to collaborate with other students, all code that you submit must be written by you (and although a great method for learning, pair programming does not abide by our standards of students submitting their own work). We also encourage using the internet to learn more about backend development, but again, any code you submit must be written by you. # FA20 Syllabus Fall 2020 ### Course Staff {% hint style="danger" %} Alanna's OH on Nov 7 Saturday will be moved to Nov 12 Thursday 1:30pm EST. {% endhint %} | Name | Title | Office Hours | Location | | -------------- | ---------- | --------------------- | ------------------------------------------------------------------------------------------- | | Alanna Zhou | Instructor | Saturdays 2-3pm EST | [Zoom Link](< https://cornell.zoom.us/j/91931470107?pwd=YzdmKytYZlpYSmZtMXJmbVlCdWtzdz09>) | | Alicia Wang | Instructor | Sundays 2-3pm EST | [Zoom Link](https://cornell.zoom.us/j/4021663130?pwd=dEdWV0dpTlJzbGJsUGFOZ1N5cmxrUT09) | | Jack Greenberg | TA | Saturdays 1-2pm EST | [Zoom Link](https://cornell.zoom.us/j/2419192491?pwd=K3R1emRVVlBWWGdtOVY2UWlnWmZmUT09) | | Yuna Shin | TA | Sundays 10am-11pm EST | [Zoom Link](https://cornell.zoom.us/j/6039891007?pwd=RjVnSHYvQVlsbmZjR1JxM0FBZjNnZz09) | | Orko Sinha | Consultant | -- | -- | | Raahi Menon | Consultant | -- | -- | | Manish Saha | Consultant | -- | -- | ### Schedule {% hint style="warning" %} This schedule is tentative and subject to changes to Cornell's Fall 2020 academic calendar. {% endhint %} | Lecture Date | Topic | Project | Project Due Date | | ------------ | --------------------------- | ------------------- | ---------------- | | 9/21 | Routes | Reddit | 9/27 | | 9/28 | Databases | Venmo (Basic) | 10/4 | | 10/5 | Relational Databases | Venmo (Full) | 10/18 | | 10/19 | Abstractions | CMS | 10/25 | | - | Break :) | - | - | | 10/26 | Containerization | Dockerize CMS | 11/1 | | 11/2 | Deployment | Deploy CMS | 11/8 | | 11/16 | Authentication | N/A | N/A | | During break | Images (Demo) | N/A | N/A | | N/A | Hack Challenge (No Lecture) | Midpoint Submission | 12/6 | | N/A | Hack Challenge (No Lecture) | Final Submission | 12/13 | | 12/14 | Hack Challenge Finale | N/A | N/A | ### Enrollment You can enroll yourself into the course like any other on Student Center. This is a 2-credit S/U course, however we permit students to enroll for 1-credit for the purpose of avoiding a credit overload petition. All students are expected to do the same work, regardless of the credits enrolled in. ### Weekly Lectures {% hint style="danger" %} All lectures and demos will be conducted online asynchronously. They will be uploaded on Youtube by every Monday and are now part of a playlist on the left called "FA20 Videos"! {% endhint %} Our lectures are broken down into two portions: a concept-based overview of the week's topic, and an implementation-based coding demo. Lectures and demos run for 10 weeks and will be posted as videos on the Cornell AppDev [Youtube channel](https://www.youtube.com/channel/UCUDPjOx1yPRGxsO6AFtBZSg). The textbook chapters also serve as a good resource alongside the lectures. ### Weekly Assignments Programming assignments are due on Sundays at 11:59pm and submitted on CMS (not Piazza!). **Late assignments will receive a flat -2 point penalty and no late submissions will be accepted after the Wednesday following the normal submission deadline. We only grant extensions to students who enroll in the course after the first lecture date.** Those who are granted extensions *must* make a private Piazza post ***and*** upload their assignments to CMS so that instructors can grade and respond in a timely manner. Each assignment will have three optional challenges to take your understanding further. * Tier I challenges are designed to be low difficulty, simple extensions of the application (estimated time: <1 hour). * Tier II challenges are medium difficulty extensions that require more forethought for a successful implementation, but are by no means out of reach (estimated time: <2 hours). * Tier III challenges are high difficulty extensions which will often require outside research and will teach you things that we do not have time to cover in this course. \ We highly encourage that you reference demos, attend office hours, and ask questions on Piazza. Along with submitting assignment source code, you will be **required** to submit a text file (`README.txt`) where you can freely comment on the week’s lecture, assignment, or anything else. You will also fill out a short questionnaire which helps us quantify feedback for improving the course. ### Grading This course is graded as S/U (pass/fail). We want all of our students to pass the course. As long as you turn in every assignment with reasonable effort and follow academic integrity, you should be on pace to pass. All assignments take into account effort with a minimum score if you submit something that shows you tried. No submission or noticeable lack of effort will receive zeros. **To pass this course, you will need >= 70** **points AND create your own app by participating in the Hack Challenge.** We have many extra credit opportunities throughout the semester on assignments (up to 6 points each) as well as user research for apps that Cornell AppDev members are working on. Also note that although not mandatory, lecture attendance is highly recommended. | Item | Weight | | ------------------- | ------- | | PA1 - Reddit | 10 | | PA2 - Venmo (Basic) | 10 | | PA3 - Venmo (Full) | 10 | | PA4 - CMS | 10 | | PA5 - Dockerize CMS | 10 | | PA6 - Deploy CMS | 10 | | Final Project | 30 | | Weekly Surveys | 10 | | **Total** | **100** | | **Passing Score** | **70** | ### Hack Challenge All AppDev courses finish with bringing members from iOS, backend, and design together to put what you’ve learned to the test and build your very own application. All students are welcome to submit app ideas and teams up to four will be formed to bring the idea to life. Leading up to the Hack Challenge, the backend course will focus on deployment so that your mobile apps can connect to a live backend running on Google Cloud servers. It goes without saying that the majority of students find the Hack Challenge the most rewarding and fun part of the course! {% hint style="info" %} Due to Spring 2020's move to online instruction, there was no Hack Challenge last semester, so students made individual final projects. We still saw awesome backend projects, so we may make this option available to students again. {% endhint %} #### Checkout Spring 2020's Final Project Award Winners! {% content-ref url="/pages/-M7OPrxqnCciFkYPZtHY" %} [Final Project Award Winners](/deprecated/final-project/final-project-winners) {% endcontent-ref %} ### Piazza We use Piazza for releasing assignments, posting links to lecture videos and transcripts, and answering whatever questions you might have. The Piazza page is linked on the left sidebar titled as `Piazza`. ### Academic Integrity As with any other course at Cornell, the code of academic integrity will be enforced in this class. While we encourage you to collaborate with other students, all code that you submit must be written by you (and although a great method for learning, pair programming does not abide by our standards of students submitting their own work). We also encourage using the internet to learn more about backend development, but again, any code you submit must be written by you. # SP20 Syllabus Spring 2020 ### Course Staff {% hint style="danger" %} On-campus office hours have been cancelled for the rest of the semester due to the University mandating virtual instruction. Office hours will resume as bookable 1-on-1 sessions with course staff over zoom once courses resume 4/6. Please refer to Piazza for further instructions. {% endhint %} | Name | Title | Office Hours | Location | | --------------- | ---------- | ---------------- | ---------- | | Alanna Zhou | Instructor | SUN 2:15-3:15pm | Rhodes 406 | | Conner Swenberg | Instructor | SUN 2:15-3:15pm | Rhodes 406 | | Alicia Wang | TA | SUN 3:00-4:00pm | Rhodes 406 | | Raahi Menon | TA | SUN 3:00-4:00pm | Rhodes 406 | | Sean Corcoran | TA | FRI 2:30-3:30pm | Rhodes 574 | | Megan Le | TA | FRI 2:30-3:30pm | Rhodes 574 | | Orko Sinha | TA | FRI 4:30-5:30pm | Rhodes 412 | ### Schedule {% hint style="warning" %} This schedule **is** updated to reflect the recent changes due to COVID-19! {% endhint %} | Lecture Date | Topic | Project | Project Due Date | | ------------ | -------------------- | ------------- | ---------------- | | 2/17 | Routes | Reddit | 3/1 | | 3/2 | Databases | Venmo (Basic) | 3/8 | | 3/9 | Relational Databases | Venmo (Full) | 4/8 | | 4/6 | Abstractions | CMS | 4/12 | | 4/13 | Containerization | Dockerize CMS | 4/19 | | 4/20 | Deployment | Deploy CMS | 4/26 | | 4/27 | Authentication | Final Project | 5/10 | ### Enrollment You can enroll yourself into the course like any other on Student Center. For convenience, our course code is **14688**. This is a 2-credit S/U course, however we permit students to enroll for 1-credit for the purpose of avoiding a credit overload petition. All students are expected to do the same work, regardless of the credits enrolled in. ### Weekly Lectures {% hint style="info" %} Cornell's course system indicates that our lectures run from 8:35-9:25pm. In the past we have run lectures from 8:30-9:30pm and typically need the full hour, so please arrive to Gates G01 by 8:30pm. {% endhint %} We will hold lectures for 10 weeks every Monday from 8:30 - 9:30pm in Gates G01 (main atrium on the basement floor). Lectures will be split into two parts: first focusing on the theoretical happenings of a backend, followed by a live demo showing the implementation of these systems with selected packages and Python. All lectures will be recorded and posted on YouTube for you to rewatch as you wish. We also have lecture transcripts for those who prefer to read at their own pace. ### Weekly Assignments Programming Assignments will be due every Sunday at 11:59pm and submitted on CMS. **Late assignments will receive a flat -2 point penalty and no late submissions will be accepted after the Friday following the normal submission deadline. We only grant extensions to students who enroll in the course after the first lecture date.** Each assignment will have three optional challenges to take your understanding further. * Tier I challenges are designed to be low difficulty, simple extensions of the application (estimated time: <1 hour). * Tier II challenges are medium difficulty extensions that require more forethought for a successful implementation, but are by no means out of reach (estimated time: <2 hours). * Tier III challenges are high difficulty extensions which will often require outside research and will teach you things that we do not have time to cover in this course. \ Referencing demos are highly encouraged as well as attending office hours and asking questions on Piazza. Along with submitting assignment source code, you will be expected to submit a text file (`README.txt`) where you can freely comment on the week’s lecture, assignment, or anything else. You will also fill out a short questionnaire which helps us quantify feedback for improving the course. ### Grading This course is graded as S/U (pass/fail). We want all of our students to pass the course. As long as you turn in every assignment with reasonable effort and follow academic integrity, you should be on pace to pass. All assignments take into account effort with a minimum score if you submit something that shows you tried. No submission or noticeable lack of effort will receive zeros. **To pass this course, you will need >= 70** **points AND create your own app by participating in the Hack Challenge.** We have many extra credit opportunities throughout the semester on assignments (up to 6 points each) as well as user research for apps we work on. Also note that lecture attendance is not mandatory, though highly recommended. | Item | Weight | | ------------------- | ------- | | PA1 - Reddit | 10 | | PA2 - Venmo (Basic) | 10 | | PA3 - Venmo (Full) | 10 | | PA4 - CMS | 10 | | PA5 - Dockerize CMS | 10 | | PA6 - Deploy CMS | 10 | | Final Project | 30 | | Weekly Surveys | 10 | | **Total** | **100** | | **Passing Score** | **70** | ### ~~Hack Challenge~~ Final Project Due to COVID-19's outbreak, the SP20 backend course will finish on a final project where you design the backend side of a new application. ~~All AppDev courses finish with bringing members from iOS, Backend, and Design together to put what you’ve learned to the test and build your very own application. All students are welcome to submit app ideas and teams up to four will be formed to bring the idea to life. Leading up to the Hack Challenge, the Backend course will focus on deployment so that your mobile apps can connect to a live backend running on Google Cloud servers. It goes without saying that the majority of students find the Hack Challenge the most rewarding and fun part of the course.~~ ### Piazza We use Piazza for releasing assignments, posting links to lecture videos and transcripts, and answering whatever questions you might have. The Piazza page is linked on the left sidebar titled as `Piazza`. ### Lecture Videos For the past few semesters, we have recorded every lecture and demo, and we will continue to for the Spring 2020 semester. Our videos from Fall 2019 are linked on the left sidebar titled as `FA19 Lecture Videos`. ### Academic Integrity As with any other course at Cornell, the code of academic integrity will be enforced in this class. While we encourage you to collaborate with other students, all code that you submit must be written by you. We also encourage using the Internet to learn more about backend development, but once again any code you submit must be written by you. # Deployment Pre-Class TODO's ### 1. Watch Pre-Lecture Video We will not be conducting a typical keynote lecture in-class, just a demo for the full hour. [**Lecture 6 Video**](https://www.youtube.com/watch?v=yZ2Av0e_oJ8\&list=PLjf6nsEcF5KOQ8nrXbKKF18NSNcpm5-HC\&index=9) {% hint style="info" %} No new demo code is required! We will be using our docker images that we previously pushed to DockerHub. {% endhint %} ### 2. Claim Google Cloud Credits We have secured $50 of Google Cloud Credits for everyone in the course to kickstart your cloud servers! To claim your credit, follow the instructions starting [here](https://google.secure.force.com/GCPEDU?cid=Lkad%2BEHrpNwTFWT4DgtvZB%2FV6oRbs9ELs00fNZTuxuigkYgPWlz%2FDdpJVJLNxCKN). Make sure to check your emails for a couple of verification steps along the way and please make sure your google account in the top right corner is set to your Cornell email before finally redeeming your code!
# PA6 Assignment Handout Contributors: Alicia Wang, Conner Swenberg ## Assignment Scope {% hint style="warning" %} **Due Date:** Sunday 11/2, 11:59pm {% endhint %} You will be deploying the docker image you created in PA5 to Google Cloud! It should have all the functionality of PA4. Your submission will only involve a README.txt file including the IP address of your deployed server. ### Helpful Links * [Docker Hub](https://hub.docker.com/) * [Google Cloud Console](https://console.cloud.google.com/) * [Installing Docker on Ubuntu 18.04](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04) * [Installing Docker Compose on Ubuntu 18.04](https://www.digitalocean.com/community/tutorials/how-to-install-docker-compose-on-ubuntu-18-04) ### 1. [Set Up Google Cloud Account](https://backend-course.cornellappdev.com/chapters/deployment/pre-class#2-claim-google-cloud-credits) ### 2. [Set Up Server Environment](https://backend-course.cornellappdev.com/chapters/deployment/demo#2-open-ssh-web-interface) ### 3. [Run Docker Container](https://backend-course.cornellappdev.com/chapters/deployment/demo#4-setup-docker-compose-file) ### 4. Submit Assignment #### 3a. Fill out your README.txt A `README.txt` file is included in the starter code for you to fill out after completing the assignment. Also note that you will not receive extra credit for Extra Credit Challenges you complete if you do not let us know for when we grade! {% hint style="warning" %} Make sure to note your server's IP address so that we can test it while grading! {% endhint %} {% code title="README.txt" %} ``` Name: Jane Smith NetID: js123 Challenges Attempted: Tier I, II, III Instance IP: ``` {% endcode %} #### 3b. Submit README.txt on CMS #### 3c. Submit Feedback Form [**Feedback Form**](https://forms.gle/ihzB543fQE2oVuGq5) To receive credit for your assignment, you **must** fill out the feedback form so that we can better understand how effectively we are teaching and how long students are spending on assignments. ## Challenges ### **Tier I** #### **Add the use of a Docker volume to isolate a production database** You can follow along the steps and motivation layed out in the demo. #### **Add the use of an environment variable file for secrets** You can follow along the steps and motivation layed out in the demo. # Deployment Demo Contributors: Alicia Wang, Conner Swenberg, Alanna Zhou ## 1**. Setup Google Cloud Server** After successfully claiming your Google Cloud credits, make your way over to the [console](https://console.cloud.google.com/) to begin setting up a server instance to deploy our code onto. We begin on the left side-menu by clicking `Compute Engine` -> `VM Instaces`. Here are the settings we used for our setup. * Name: demo * Region: us-east-1 * Machine configuration * Machine family: General purpose * Machine type: g1-small * Boot disk: Ubuntu 18.04 * Firewall: Allow HTTP traffic After setting up your machine, you should see it in your `VM Instances` console page as a singular row. ## **2. Open SSH Web Interface** On the right side of your newly created VM Instance row (within the dashboard), there should be a button titled `SSH`; Click it. Doing so will prompt a new window pop-up and a command line interface. You are now "inside" your cloud server and can treat its command line analogously to your own computer. Now that we have control within the server, we can begin to set up our machine to run your application code, starting with installing Docker and Docker Compose. ## **3. Install** **Docker** **and** **Docker Compose** ### Docker Follow Step 1 of this [guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04). Continuing with Step 2 is optional and only gives us the convenience of running commands without preprending the phrase `sudo` before everything relating to Docker. It increases your permissions on the server to that of a `root` user, which can do things like run Docker containers without manual override on each command. We generally recommend just skipping this step and instead make use of the `sudo` keyword on relevant commands, which grants you the ability to just run that singular command under root privileges. ### Docker Compose Follow Step 1 of this [guide](https://www.digitalocean.com/community/tutorials/how-to-install-docker-compose-on-ubuntu-18-04#step-1-%E2%80%94-installing-docker-compose). ## 4. Setup Docker Compose File Just as we demonstrated in the previous chapter, we can streamline our use of Docker containers by specifying a `docker-compose.yml` file. We can use the same file used in the last demo which is already pointing to our image pushed to DockerHub. To create a new file from the SSH'd command line, we will use Linux's built in `nano` tool. To create our new file, type: ```python >>> nano docker-compose.yml ``` This will pull up a new interface for you to edit the file. Specify the file the same as the one used previously. ```yaml version: "3" services: demo: image: username/image ports: - "80:5000" ``` Take note that now our `ports` mapping goes from `80` to `5000`. We want HTTP requests that hit our server's IP address to be routed into our container and the reserved port for HTTP is port 80. This is not necessarily a requirement because we could just as easily build our frontend to send requests to `xxx.xx.xx.xxx:5000` but this is just more verbose than simply sending requests directly to the IP. ## 5. Run Docker Container As demonstrated last chapter, we can run our Docker containers with: ```python >>> sudo docker-compose up -d ``` Note that now we need to include the `sudo` flag to utilize root permissions. Running this command is functionally equivalent to: ```python >>> sudo docker run -it -p "80:5000" username/image ``` However, now we have the ability to run the container in a **detached** mode (the `-d` flag), which means quitting out of our SSH window will not terminate the running of our container. Because this server does not have our indicated image tag locally, Docker will automatically search DockerHub for a match and download it onto this machine to run. After **pulling** our image from DockerHub, you will see Docker downloading the **layers** of our application from the ground up (includes our base python installation for the container as an example). You can confirm the active running containers on your server (and personal computer) with: ```python >>> sudo docker ps ``` This will output identifying information like a unique id and the image tag. This confirmation solidifies that our code is running on an external server. Our final test to see if our deployment was successful is to visit our IP address indicated in Google Cloud's VM Instances console to test our API with real requests. ## **Other Tips** ### **Redeploying after updating code** Let’s say we want to make a local change to our code, such as changing a route response. {% code title="app.py" %} ```python @app.route("/") def hello_world(): return "Hello World!", 200 ``` {% endcode %} We can rebuild the image and then push it to Docker Hub: ```python >>> docker build -t username/image . >>> docker push username/image ``` Note that a lot of the image layers will say “Layer already exists”. Docker can recognize unchanged layers and won’t re-push them. Because the application includes a lot of libraries like Python and OS libraries, a lot of time could be wasted if Docker re-uploaded them. Check that the image was successfully pushed on Docker Hub. The latest tag should say that it was updated a few seconds ago. Go back to the GC server and repull the new image (similar to Git). You must kill the currently running container before redeploying the newer image because the port is already in use. ```python >>> sudo docker-compose down >>> sudo docker pull username/image >>> sudo docker-compose up -d ``` When you refresh at the server IP address, you should see `Hello World!` ### Using Docker volumes In our current setup, whenever we redeploy our application code, a new `.db` file comes along with it in the bundled image. This means we wipe our deployed `.db` file's contents with each redeployment which is a problem. To solve this, we need a way to maintain our sqlite3 database outside of the scope of the Docker container on our external server. We do this with a Docker **volume**. Just like how a port mapping within `docker-compose.yml` takes a port outside the container to be used as a different port inside the container, we can make a volume mapping to link a file outside of the container to be used as a file inside the container. First we need to decide where to place this file: the same level as our `docker-compose.yml` will do just fine. Add a new file with: ```python >>> touch todo.db ``` Later we will need to know the exact path to this file, so you can discover by `cd ..` a couple times to see how the remote server's directories are structured. To help you get this path correct, here is the default directory structure for Google Cloud's VM Instances: ```python / |-home/ |-/ |-docker-compose.yml |-todo.db ``` Note that `` will be your netid by default when making your own VM Instance. Therefore, our full exact path to our new database file is: `/home//todo.db`. We also need to know the exact path to our database file within the container which should be `/usr/app/todo.db` given how our Dockerfile previously defined this new directory for our application code. To connect the volume, update your compose file like so: {% code title="docker-compose.yml" %} ```python version: '3' services: demo: image: username/image ports: - "80:5000" volumes: - /home//todo.db:/usr/app/todo.db ``` {% endcode %} Now whenever we redeploy, the contents of our `todo.db` file locally have no impact on the remote server's `todo.db` because the file outside the container is untouched. Now we can safely update our application code without changing our database's contents. ### **Using environment variables** If you’re implementing authentication or you have some sort of client secrets, you may have a local environment (.env) file that you don’t want pushed to any publicly available repository like on Github. Let’s test using an environmental variable locally before we use it on deployment: {% code title="secrets.env" %} ```python export SECRET_KEY=helloworld ``` {% endcode %} Source `secrets.env` into the environment, i.e. load the variables into the command prompt. The `export` is necessary to make the environment variables available to subprocesses as well - otherwise an error would be raised in Python when the app tries to retrieve `SECRET_KEY`. ```python >>> source secrets.env ``` We can update our `app.py` to make use of this environment variable, such as returning the secret as a response. Don’t actually do this in any of your own apps! {% code title="app.py" %} ```python import os ... @app.route("/") def hello_world(): return os.environ["SECRET_KEY"], 200 ``` {% endcode %} Now before we rebuild and push our code to Docker Hub, lets ignore any environment files from packaging using a `.dockerignore` file: ```python __pycache__ venv *.env ``` Notice that before we were ignoring our python cache and virtual environment folder to limit pushing useless files. Another trick is to use `*.env` to encompass all files with the `.env` extension. Now we can safely rebuild and repush our image to Docker Hub: ```python >>> docker build -t username/image . >>> docker push username/image ``` We can then modify our `docker-compose.yml` on the GC server to include a corresponding `secrets.env` file: {% code title="docker-compose.yml" %} ```yaml version: "3" services: demo: image: username/image env_file: - secrets.env ports: - "80:5000" ``` {% endcode %} Create the same secrets.env file on the Google Cloud server without the export keyword. Docker uses a special format of env files to allow this. {% code title="secrets.env" %} ```python SECRET_KEY=helloworld ``` {% endcode %} Pull the newest image, and when you start the server, now the `/` path should return `helloworld`. # Final Project (Spring 2019) Contributors: Conner Swenberg {% hint style="warning" %} **Due Date:** Sunday 5/10, 11:59pm {% endhint %} ## Introduction Throughout the course, you have practiced building backends to fulfill some predefined API specification. Know that you have experience constructing routes, designing a database, and deploying application code, you are challenged to create a backend completely of your own design. Traditionally, we have brought students in all of our courses together to build a fully functional application. Unfortunately, this semester will miss out on that opportunity, but you will still experience the same level of challenge. This assignment is worth **30 points** and is mostly graded on completion. We want you to demonstrate your new skills while having fun designing a new kind of application. There are many extra credit opportunities as well which draw from previous challenges in the course. Additionally, there are 3 challenge competitions for fun and will not count towards extra credit, just pride. ### Team Requirements This will be a partner project, but those who wish to roll solo can do that as well. Groups of 3 or more are not allowed. Collaborating on a codebase is difficult without a tool to help you. We will show you how to use Git, an industry standard for collaborating on software. We strongly recommend students to share responsibility in designing and coding in all parts of the project (i.e. routes, database, deployment) and intend more for this project to be like pair-programming. ## Project Requirements All of the following are requirements for the project and will be graded on a complete/incomplete basis. We believe each is important for your development and breadth of experience. ### 1. GitHub Repository You will not be submitting code on CMS for this assignment; you will instead make a public Github repository and submit the link. Even those who are doing the project solo will be required to make a Github repository; those who collaborate will only need to have one person create the repository to work on. We also recommend you make your repository on a traditional public account versus Cornell's private enterprise option so that you can more easily share this project with recruiters and anyone else who might want to look! ### 2. Database You have been exposed to both the `sqlite3` and `SQLAlchemy` packages for designing database tables and hopefully notice the many benefits of using an ORM. You are required to use `SQLAlchemy` to build **at least one one-to-many relationship and at least one many-to-many relationship**. Additionally, you are required to implement some form of **nested serialization on related fields** (like how we did in PA4). Additionally you must make your implementation meaningfully different than applications we have built in the assignments so far. This means if you want to build an app overlapping with a to-do, forum, payments, or course management functionality, you should first check with the course instructors to make sure your application will be challenging you enough to design something new. **If you build a to-do, forum, payments, or course management application without talking to us first, you risk getting a zero on the final project.** ### 3. Routes You have built some set of routes in each of the first four assignments and should be accustomed to common design patterns. You must implement **at least one route for each operation archetype: create, retrieve, update, delete** (CRUD). We encourage you to use many of the abstractions shown throughout the course for clean code. For example, the use of a DAO is encouraged for your routes, but not an official requirement. ### 4. Deployment You have learned how to containerize application code, ssh into a cloud server, and run your application accessible from a public IP. You must **containerize your application with Docker** and **provide us a public server IP** to interact with your application. You should have more than enough credits to keep your server running for over a month so do not take your application down until we have submitted your grade for the final project. ### 5. API Specification & Documentation We want you to **document how people should interact with your application with an API specification** (like we have been providing for you on each assignment). We recommend using [markdown](https://daringfireball.net/projects/markdown/syntax) for easy templating and supported by Github with `.md` files. Just add an `api.md` file or describe your routes within the `README.md` file made by default with a new git repository. Writing a clear API specification is also important for helping us grade your final project when trying to interact with your application ourselves with Postman to test it. Feel free to use our styling for API specification or think of your own way to convey the essential information. Additionally you can try popular tools like [Swagger](https://swagger.io/) and [Open API](https://www.openapis.org/) for documentation. ## Extra Credit Challenges If you're on the edge of not passing this course or just want to have fun learning something new, we recommend exploring the following challenges. Successful completion of **each will be worth 3 points of extra credit** (equivalent to a Tier III challenge). ### 1. Draw data from an external API For the context of your application, you may want to populate your database with information on some different service. Your challenge is to integrate with some other backend and transform its data into your own application. ### 2. Integrate with a communication API For many applications, automating communication with users and other parties is a huge value-add. In previous assignments we offered integrating with SendGrid's email API as a Tier III challenge for example. ### 3. Support raw file storage Many applications demand some form of media that needs to be managed by the backend (i.e. images, video, audio, etc.). In PA4 we offered integrating with Amazon Web Services S3 micro-service as a Tier III challenge. Google Cloud also has its own version called "Cloud Storage". ### 4. Add user authentication Applications that are serious about professional use need some authentication system to preserve information security. Chapter 7 covers many authentication schemes for you to draw inspiration from. ## Competition Challenges We want to celebrate the amazing applications you will build with some fun competition. These challenges will not be worth any extra credit or official prize and there will likely be many fantastic backends that do not get an award due to the number of submissions. Each group is capped at winning only one challenge. ### 1. Best Documentation We cannot understate the value of high quality documentation which can include API specification, comments within code, and the styling/naming/layout of code in general. Anything that contributes to the ease for someone new to understand your application at any level ranging from high and abstract to low and precise is a marking of a seasoned developer. This award will go to whoever demonstrates the best documentation standards and code cleanliness. ### 2. Best Application Design While we did not deeply discuss the best design principles for applications, we subtly dropped many tips throughout the course. Pretty much all of the design patterns taught in this course apply to all sorts of applications and we want to see how you apply your knowledge to a more complex application than our boilerplate examples. This award will go to whoever demonstrates quality database design, choice of route interfaces all in the context of a creative application. ### 3. Most Creative Application Although there was no app to go with your backend this semester, we still love to see backends that envision a greater idea to fulfill. This award will go to whoever demonstrates a refreshing idea that serves not only a need in the world but is also creative! ### 4. Best Security Security is one of the most important aspects of backend, so much that DevOps is becoming DevSecOps. Developers should always strive to think about the best ways to secure their applications, and taking advantage of our authorization lectures is a good step forward! This award will go to whoever takes advantage of features of our authentication lecture! ### 5. Best Overall Backend Combining all of the aspects of the previous three challenges, the Best Overall Backend will demonstrate high levels of achievement in each category. # Final Project Award Winners Conner Swenberg, Alanna Zhou ## Spring 2020 Congrats to everyone who finished their final projects! You guys made it! Everyone did an amazing job--it was honestly quite hard to choose the awards this semester. Now we'll announce the winners of the Spring 2020 Backend Awards! ### 1. Best Documentation Congrats to **Helen Liang** and **Hilarie Sit** for creating documentation with Swagger API!! Props to making such an extensive backend--runner up for Best Application Design. {% embed url="" %} ### 2. Best Application Design Congrats to **Allison Hsu** and **Ava Tan** for best application design! We received multiple Slack / Group messaging projects, but you guys did such an awesome job for extending your application with so many features! {% embed url="" %} ### 3. Most Creative Application Congrats to **Gonzalo Gonzalez-Pumariega** for creating such a unique backend with really extensive logic for a game application. Kudos to all of those intense game score calculations! {% embed url="" %} ### 4. Best Security Congrats to **Haiying Weng** for making the only backend that integrated authorization with a token! We love a security conscious backend!! {% embed url="" %} ### 5. Best Overall Backend Congrats to **Tewodros Mitiku** for creating such a cool and unique backend (with Swagger docs!) -- instructors Conner and Alanna tested out this Spotify and Sendgrid API integrated backend, and were pleasantly surprised to find an email in our inboxes that showed our commonly listened to songs! {% embed url="" %}