Nine Essential Tips for Using Alibaba Cloud OSS

Reading and Writing Files on OSS

To operate on files in OSS, you do not use traditional functions like fopen() or fclose(). Instead, you use RESTful HTTP requests: PUT to write, GET to read, HEAD to get file attributes, and DELETE to remove a file.

After configuring your OSS Python development environment, you can use a declared storage object (e.g., my_store) to create and write a new file (called an Object in OSS). Example code:

res = my_store.put_object(bucket_name, object_name, input_content, content_type)

The content_type should be the MIME type defined by the HTTP protocol for the file type (e.g., image/jpeg for JPG, audio/mpeg for MP3). Setting this correctly helps other web applications use your OSS files properly.

To read an existing file:

res = my_store.get_object(bucket_name, object_name)

To get file attributes:

res = my_store.head_object(bucket_name, object_name)

To delete a file:

res = my_store.delete_object(bucket_name, object_name)

With these four simple functions, you can easily migrate traditional file-system applications to the OSS cloud storage platform.

Hotlink Protection with Signed URLs

OSS provides excellent network bandwidth, making it ideal for hosting images, music, and videos. To prevent unauthorized hotlinking, use signed URLs.

First, set your Bucket's permission to private, so all requests require authentication. Then, dynamically generate a signed URL based on the operation type, target Bucket, target Object, and expiration time. Authorized users can perform the specified operation before the URL expires.

Python example to generate a signed URL:

url = my_store.sign_url(method, bucket_name, object_name, timeout=60)

Here, method can be PUT, GET, HEAD, or DELETE; timeout is in seconds. The generated URL looks like:

http://storage.aliyun.com/sharedata/lingyun.jpg?OSSAccessKeyId=y6h7nbcothehvcp7jlnwmrw9&Expires=1335084740&Signature=LZeqnHSo5WkDNWKffKDgQBXR6fY=

This method effectively protects your OSS data from unauthorized access.

Conditional Data Transfer

OSS supports four conditional transfer parameters. Data is only transferred if the Object's attributes meet the client's specified conditions:

• If-Modified-Since
• If-Unmodified-Since
• If-Match
• If-None-Match

If-Modified-Since is a common HTTP parameter. The server checks if the data is newer than the provided timestamp. If not, it returns the data; if it is, it returns a 304 status code, telling the client to use its local cache. This reduces network traffic and server load.

If-Unmodified-Since works oppositely: data is returned if the content is unchanged; otherwise, a 304 is returned. These two can be combined to define a time window. Example:

headers = {}
headers['If-Modified-Since'] = "Sun, 22 Apr 2012 09:06:23 GMT"
headers['If-Unmodified-Since'] = "Sun, 22 Apr 2012 09:16:23 GMT"
res = my_store.get_object(bucket_name, object_name, headers)

If-Match and If-None-Match are similar but use the content's ETag (usually an MD5 hash) as the condition. Using these parameters wisely can save significant bandwidth and cost.

Implementing Folder Functionality on OSS

While OSS uses Buckets and Objects, you can logically implement folders. Convention: any Object ending with / is treated as a folder. For example, folder is a file, folder/ is a folder, and folder/file.txt is a file inside that folder. All three are independent Objects in OSS.

To list files within a folder, use the List Objects (Get Bucket) interface with these parameters: prefix, marker, delimiter, and max-keys.

Assume a Bucket named mydata contains:

lingyun.doc
folder/
folder/file1.txt
folder/file2.txt
folder/file3.txt
folder/image/
folder/image/test.jpg

To view it as a traditional file system (showing only lingyun.doc and folder/), set delimiter='/':

res = my_store.list_objects("mydata", delimiter='/')

OSS returns an XML response with a key for lingyun.doc and a common prefix for folder.

To list files inside folder, set prefix='folder/':

res = my_store.list_objects("mydata", prefix='folder/', delimiter='/')

This shows three files (file1.txt, file2.txt, file3.txt) and a subfolder (image/).

max-keys limits the number of results per request (default 100, max 1000). If a folder has more than 1000 files, use marker to start listing from a specific file. Example:

res = my_store.list_objects("mydata", prefix='folder/', marker='folder/file1.txt', delimiter='/', maxkeys='1')

This returns only folder/file2.txt. Using these parameters, you can build applications like Dropbox.

Resumable and Concurrent Downloads

Resumable download records the last received byte position and asks the server to send the remaining data. Use the HTTP Range header with OSS.

To get the first 5 bytes:

headers = {}
headers['range'] = "bytes=0-4"
res = my_store.get_object(bucket_name, object_name, headers)

To get 3KB starting from byte 1024:

headers = {}
headers['range'] = "bytes=1024-4095"
res = my_store.get_object(bucket_name, object_name, headers)

Concurrent download is simple: split the Object into chunks, start multiple threads to download each chunk, and combine them when done.

Note: When using the range header, the server returns HTTP status 206 (Partial Content).

Concurrent Upload of Large Files

Uploading large files over the internet can be unstable. OSS's Multipart Upload mode splits a large file into smaller Parts, uploads them independently (allowing concurrency), and combines them on the server.

Using the SDK simplifies this. For example, to upload with 10 threads:

res = my_store.multi_upload_file(bucket_name, object_name, thread_num=10)

Refer to the OSS API documentation for implementation details.

Deleting a Bucket with Many Objects

OSS prevents deletion of a non-empty Bucket. To delete thousands of files efficiently, first list the Objects, then use the batch delete interface. Python SDK example:

object_list = []
while True:
    object_list = my_store.list_objects(bucket_name)
    if len(object_list) != 0:
        my_store.batch_delete_objects(bucket_name, object_list)
    else:
        break

This deletes tens of thousands of files with just a few requests.

Adding Custom Headers to Objects

You can attach custom metadata to files (e.g., photo capture time, article author, song album) without downloading the entire file. Use the x-oss-meta- prefix in PutObject.

Example to add an author:

headers = {}
headers['x-oss-meta-author'] = 'obama'
res = my_store.put_object(bucket_name, object_name, headers)

When retrieving the Object, the response header will include:

X-oss-meta-author: obama

Debugging Code with OSS

If an HTTP request to OSS is invalid or non-compliant, OSS returns an error code and detailed XML message. For any non-2XX HTTP status, OSS provides an XML body explaining the failure.

Example for a 403 Forbidden error:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>AccessDenied</Code>
  <Message>Access denied.</Message>
  <RequestId>17baec8b-1a0e-8dad-4a6e-343b4d8450dc</RequestId>
  <HostId>storage.aliyun.com</HostId>
</Error>

The RequestId is a unique UUID for the request. If you cannot resolve the issue, provide this ID to OSS technical support.