Django Tutorial: How to Upload Large Files
This tutorial will show you how to upload large files with Django. We will store these files on an S3 bucket using Digital Ocean Spaces.
Configure a Django Project
Install the required packages required for using S3:
pip install django-storages boto3Add the following to settings.py. You will need to add the credentials of your bucket in here.
AWS_ACCESS_KEY_ID = ""
AWS_SECRET_ACCESS_KEY = ""
AWS_S3_REGION_NAME = ""
AWS_S3_ENDPOINT_URL = ""Start a Django app:
python manage.py startapp uploaderAdd the app to the INSTALLED_APPS in settings.py
File Model
Create a model to store the uploaded file relationship:
class UploadFile(models.Model):
file = models.FileField()
uploaded_date = models.DateTimeField(auto_now_add=True)Run python manage.py makemigrations and python manage.py migrate
Views
The first thing to do is created a view that returns a signed URL. The boto3 package can be used to generate temporary signed URLs that allow you to upload files directly to your S3 bucket. Instead of uploading files through the Django project, the files will now be uploaded directly to the signed URL.
import json
import boto3
from django.conf import settings
from django.http import JsonResponse
from django.views import generic
class SignedURLView(generic.View):
def post(self, request, *args, **kwargs):
session = boto3.session.Session()
client = session.client(
"s3",
region_name=settings.AWS_S3_REGION_NAME,
endpoint_url=settings.AWS_S3_ENDPOINT_URL,
aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
)
url = client.generate_presigned_url(
ClientMethod="put_object",
Params={
"Bucket": "media",
"Key": f"uploads/{json.loads(request.body)['fileName']}",
},
ExpiresIn=300,
)
return JsonResponse({"url": url})In the view we are creating an S3 session using our credentials. We use the client.generate_presigned_url method to create a signed URL. Notice that you need to pass in the name of the S3 bucket for where the file will be stored. I am hardcoding this to media but you could also use a value from the Django settings. I am also specifying the Key of the uploaded file to be the file's name.
The view returns a JSON response with the signed URL. We will use this url in the frontend (in this case just plain HTML and JS).
Next we will create a view to handle the form submission and create the UploadFile record. In this view we're also adding context just to include all of the uploaded files so we can render them in the frontend.
from django.urls import reverse
from .models import UploadFile
class UploadView(generic.CreateView):
template_name = "upload.html"
model = UploadFile
fields = ['file']
def get_success_url(self):
return reverse("upload")
def get_context_data(self, **kwargs):
context = super(UploadView, self).get_context_data(**kwargs)
context.update({
"uploads": UploadFile.objects.all()
})
return contextAdd both of the views to the project urls.py:
from django.contrib import admin
from django.urls import path
from uploader import views
urlpatterns = [
path('admin/', admin.site.urls),
path('', views.UploadView.as_view(), name='upload'),
path('signed-url/', views.SignedURLView.as_view(), name='signed-url')
]Register the templates folder in the TEMPLATES section inside settings.py. In Django version 3.2 and above you can set it using the following:
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [BASE_DIR / "templates"],
...
},
]Create a new templates folder and inside it create the template upload.html:
<link href="https://unpkg.com/tailwindcss@^2/dist/tailwind.min.css" rel="stylesheet" />
<div class="max-w-4xl mx-auto py-5">
<h1 class="text-2xl text-gray-800 mb-3">Upload a Large File</h1>
<form method="post" id="fileForm" enctype="multipart/form-data">
{% csrf_token %} {{ form }}
<div class="mt-3">
<div class="mt-3">
<div class="shadow w-full bg-gray-100">
<div
id="progressBar"
class="bg-blue-500 text-xs leading-none py-1 text-center text-gray-800"
style="width: 0%"
>
0%
</div>
</div>
</div>
<div class="mt-2">
<h3 id="status"></h3>
</div>
</div>
<hr class="mt-5" />
<button
type="submit"
id="submitBtn"
class="mt-5 rounded shadow-md px-3 py-1 text-lg text-white bg-blue-500 hover:bg-blue-600"
>
Submit
</button>
</form>
<hr class="mt-5" />
<div class="mt-5">
<h3 class="text-lg text-gray-800 mb-3">Upload History</h3>
{% for upload in uploads %}
<div class="py-5 px-3 border border-gray-200 bg-gray-50">{{ upload.file.name }}</div>
{% empty %}
<p class="text-gray-800">No uploads</p>
{% endfor %}
</div>
</div>
<script>
// TODO add JavaScript in here
</script>Then add the following JavaScript inside the <script></script> tags:
let submitting = false;
let file = null;
function setIsSubmitting(val) {
submitting = val;
}
function setFile(val) {
file = val;
}
_("id_file").addEventListener("change", (event) => {
setFile(event.target.files[0]);
});
_("submitBtn").addEventListener("click", (event) => {
handleSubmit(event);
_("submitBtn").disabled = true;
});
const handleSubmit = async (event) => {
setIsSubmitting(true);
const signedUrl = await getSignedUrl();
try {
uploadFile(signedUrl);
} catch (err) {
setIsSubmitting(false);
console.log(err);
alert("There was an error uploading your file.");
throw err;
}
setIsSubmitting(false);
};
const getSignedUrl = async () => {
const body = {
fileName: file.name,
fileType: file.type
};
const response = await fetch("{% url 'signed-url' %}", {
method: "POST",
body: JSON.stringify(body),
headers: { "Content-Type": "application/json", "X-CSRFToken": "{{ csrf_token }}" }
});
const { url } = await response.json();
return url;
};
function _(el) {
return document.getElementById(el);
}
function uploadFile(signedUrl) {
var formdata = new FormData();
formdata.append("file", file);
var ajax = new XMLHttpRequest();
ajax.upload.addEventListener("progress", progressHandler, false);
ajax.addEventListener("load", completeHandler, false);
ajax.addEventListener("error", errorHandler, false);
ajax.addEventListener("abort", abortHandler, false);
ajax.addEventListener("loadend", loadendHandler, false);
ajax.open("PUT", signedUrl);
ajax.setRequestHeader("Content-Type", file.type);
ajax.setRequestHeader("x-amz-acl", "public-read");
ajax.send(formdata);
}
async function submitForm() {
_("fileForm").submit();
}
async function loadendHandler(event) {
_("submitBtn").disabled = false;
await submitForm();
}
function progressHandler(event) {
var percent = Math.round((event.loaded / event.total) * 100);
_("progressBar").style.width = `${percent}%`;
_("progressBar").innerText = `${percent}%`;
_("status").innerHTML = percent + "% uploaded... please wait";
}
function completeHandler(event) {
_("status").innerHTML = event.target.responseText;
_("progressBar").style.width = 0;
_("progressBar").innerText = "0%";
}
function errorHandler(event) {
_("status").innerHTML = "Upload Failed";
}
function abortHandler(event) {
_("status").innerHTML = "Upload Aborted";
}There's a bit of JavaScript we're using to make things feel a bit smoother so let's walk through it.
We are listening for changes to the file input field. Once a file is selected, we store it inside the file variable by using the setFile function.
_("id_file").addEventListener("change", (event) => {
setFile(event.target.files[0]);
});We are also listening for the form submit button to be clicked. When the button is clicked we call the handleSubmit function which starts the process of uploading the file to the Digital Ocean Space.
_("submitBtn").addEventListener("click", (event) => {
handleSubmit(event);
_("submitBtn").disabled = true;
});The handleSubmit function starts by requesting a signed URL by sending a request to the signed-url view. The signed URL is then passed into the uploadFile function which sends an XMLHttpRequest with the file attached to it.
There are also a few event listeners added to the ajax request so that we can receive updates for the progress of the upload, as well as when it is completed, if it is cancelled or if an error occurred.
Connect to your Digital Ocean Space
In your Digital Ocean account, click on Spaces in the sidebar and fill in all the details to create the Space. You do not need to configure it as a CDN.
Once created, you'll be able to see the endpoint URL of your space.
Set this URL in your settings.py:
AWS_S3_ENDPOINT_URL = "https://tutorial.fra1.digitaloceanspaces.com"The region is also specified in this URL. In my case it is fra1. Set this value in your settings.py:
AWS_S3_REGION_NAME = "fra1"CORS Settings
The Space will block requests from localhost. We need to add CORS rules to allow specific origins to upload files to it. In production you should add only the production origins but for development we are just going to add an accept-all rule:
Click Save Options.
Now you will need API keys to connect to the Space. In the sidebar click on API. On the next page under "Spaces access keys" click on generate new key and give it a name. You will be given an access key and a secret key. Copy both and set them as the values in settings.py:
AWS_ACCESS_KEY_ID = "<your_access_key>"
AWS_SECRET_ACCESS_KEY = "<your_secret_access_key>"Conclusion
Head to your localhost to test the upload form. Select a file and click upload. You should see a progress bar load all the way to 100% and then the form being submitted to the Django backend.
To confirm that everything worked, check your Digital Ocean Space to see that the file has been uploaded.
On the Django frontend you should also see the newly uploaded file display under Upload History when the page reloads.
You can find the code for this project on GitHub