QNAP Systems, Inc. - Network Attached Storage (NAS)

Language

Support

Hosting S3 and OpenStack-compatible object storage services in QTS

Introduction

Overview

QNAP Object Storage Server (OSS) App enables the QNAP Turbo NAS to support data access using S3 and OpenStack-compatible object storage protocols, which are now the most popular standards for accessing cloud storage. OSS App complements the accessibility and compatibility of your Turbo NAS, allowing you to enjoy the growing number of versatile applications designed for cloud object storage, including backup & archiving, content delivery, and document management. As cloud storage continues to grow in popularity, most new applications are designed for cloud storage, allowing you to use a Turbo NAS to serve your applications for production and for testing. You can also migrate your data from public cloud storage to your Turbo NAS without rewriting your applications.

OSS App offers truly unified data access, allowing you to access a file using object storage protocols and also access an object using network file sharing protocols (including SMB/CIFS, NFS, FTP, and WebDAV). For example, you may use SMB/CIFS over local networks to access and manage your files on the Turbo NAS, and provide data access over Internet to the same set of files using object storage protocols. You can also control which users on the Turbo NAS can access data via object storage protocols and have multiple access keys for each user. You can also control which shared folders can be mapped to an object storage account and which users can access the storage account. By integrating network file access and object storage, OSS App makes hosting object storage services in QTS easy and secure.

Feature highlights

  • Supports S3 and OpenStack-compatible object storage protocols
  • Supports large objects by dynamic and static manifest for OpenStack and multi-part for S3
  • Supports unified storage access to files via object protocols and network file protocols
  • Supports access control at user and shared folder level
  • Allows domain users to access object storage
  • Supports up to 20 access key pairs per NAS user
  • Supports access via stand-alone object storage server or QTS web server virtual host
  • Supports private and public access controls for containers
  • Supports displaying object storage access statistics on dashboard
  • Supports logging administration operations for auditing and tracing configuration changes

Please note: The OSS App is only compatible with x86-based Turbo NAS running QTS 4.1 or above.

Object storage

Data organization

Object storage is a new way to organize data by addressing and manipulating discrete units of data called objects. Each object, like a file, is a stream of user-defined binary data. However, unlike files, objects are not organized in a hierarchy of folders and are not identified by its path in the hierarchy. Each object is associated with a key made of a string when created, and you may retrieve an object by using the key to query the object storage. As a result, all of the objects are organized in a flat name space. That is, there are no folders within a folder, and there is no operation for moving a folder. This organization eliminates the dependency between objects, but retains the fundamental functionality of a storage system: storing and retrieving data. Huge scalability and high availability for storage can be achieved with this data organization.

There are two basic levels separate the name space of objects: storage account and container. Each storage account represents an accounting and billing identity, which is similar to a storage volume. And containers can be created in a storage account to provide a primitive solution for grouping objects and to provide multiple name spaces. As mentioned, inside a container, there are no other containers.

Data access

The protocol to access object storage is the RESTful API, which is based on HTTP/HTTPS, and can be used in most network environments. The object storage API of OSS is compliant with both two object storage APIs: S3 and OpenStack. S3 is the Amazon Web Service Simple Storage Service, which is one of the leading and most popular cloud storage services.
OpenStack object storage (previously called Swift) is a leading open source project endorsed by many vendors, including IBM, HP, and Red Hat. OpenStack is also a promising solution not only for hosting public cloud services but also for private clouds. With the support for both APIs, OSS allows you to leverage a wide range of applications and migrate data between the two APIs.

However, because object storage is an evolving technology, OSS does not support all of S3 and OpenStack’s features. Please refer to the supported feature set of API in this document.

For more information about the object storage APIs, please visit the following links:

Unified storage

To simplify the administration of object storage services, OSS assumes the following mapping between the storage entities in object storage and network file storage:

  • Mapping storage accounts to shared folders
  • Mapping containers to the top-level folders in a shared folder
  • Mapping objects to files

For example, when there is a shared folder a corresponding storage account exists. And when you create a top-level folder by SMB/NFS in a shared folder you also create a container for the storage account in OSS. With the unified structure, a file in a shared folder is addressable by object storage protocols using the path of the file as its object key. If you change the path of a file by renaming or moving the file via network file protocols, its object key will also be changed correspondingly. A folder in file system will be accessed as a zero-byte object in the object storage, also with the path as its object key.

Because your data can be managed and accessed by both object and file operations, you are free to choose the access methods that fit your needs.

Please note: Most object storage client applications are designed for general S3 and OpenStack services and for better compatibility restrict users from using non-ASCII characters as the name of storage accounts, containers, and objects. To ensure that OSS can work with the client applications, please only use ASCII characters when naming the entities (e.g. storage accounts and containers) that will be accessed by object storage services.

Access control

Object storage protocols are accessed using access keys that replace conventional usernames and passwords to reduce the exposure of user credentials for better security. For added security, users cannot access data via object storage protocols by default – you need to explicitly add users from local accounts of your Turbo NAS or domain servers to the user list of OSS, and create access keys for them.

Because object storage is designed for Internet access, shared folders cannot be accessed by object storage protocols by default for security reasons, and you need to explicitly permit any storage accounts (shared folders) to be accessed by OSS users. Even if a user has the permission to access a shared folder over network file protocols (like FTP) they cannot access the corresponding storage account until you add them to the permission list for that storage account in OSS.

The owner of uploaded objects will be assigned to the user having the access keys to do the operation. However, for simplicity, the advanced permissions (ACLs) for files and folders will not be applied to object storage service.

Large objects

OSS supports large file access using Multipart Upload for S3-compatible APIs and Large Objects for OpenStack storage API.

S3 Multipart Upload allows you to upload a single object as a set of parts. After all of these parts are uploaded, the data will be presented as a single object.

OpenStack Large Object is comprised of two types of objects: segment objects that store the object content, and a manifest object that links the segment objects into one logical large object. When you download a manifest object, the contents of the segment objects will be concatenated and returned in the response body of the request. OSS supports two types of OpenStack Large Object: Static Large Object (SLO) and Dynamic Large Object (DLO).

With this feature, you can break your larger objects into chunks and upload a number of chunks in parallel. It can improve your upload speed by taking advantage of parallelism. If the upload of a chunk fails, you can simply restart it without retransmitting the whole object.

Please note: OSS stores each chunk (S3 part or OpenStack segment) as a file. You cannot use non-object storage protocols to access the large object.

Using OSS App

Overview

OSS is developed based on OpenStack Swift project, and it is optimized for QTS in terms of performance and simplicity. You may conduct the following steps to start accessing your Turbo NAS like how you access cloud storage services:

  1. Install the OSS App from the QTS App Center
  2. Add Turbo NAS users to the OSS user list
  3. Create access keys for OSS users
  4. Configure the access permissions for storage accounts
  5. Use S3 or OpenStack-compatible tools to access data using object storage protocols

Installing and launching OSS App

To install OSS on your Turbo NAS, sign into the QTS web interface as an administrator. Then go to "App Center", find "Object Storage Server" and click "Add to QTS". After downloading and installing the OSS, you will find its icon on your QTS desktop and Quick Start menu. Click "Object Storage Server" to open its management interface.

Main menu

You can switch between main management user interface web pages by clicking on the left-hand menu items. There is a summary of the functions of each menu item below:

Dashboard:
(1) Viewing statistics of object storage
(2) Turning on/off the object storage service
(3) Viewing the URL of service endpoints

User:
(1) Managing who can use the object storage service to access data
(2) Managing user access credentials (access keys)

Storage Account:
(1) Managing which storage account can be accessed by which users
(2) Creating and deleting storage accounts
(3) Viewing the capacity usage of each storage account

Container:
(1) Creating and deleting containers
(2) Enabling anonymous access for containers
(3) Viewing the capacity usage of each container

Service Endpoint:
(1) Choosing to use the built-in server or QTS web server
(2) Configuring the service port of the built-in server
(3) Configuring the virtual host name for the service

Dashboard

OSS provides a dashboard for showing the statistics and the endpoints of the object storage service. You can enable or disable the service here. The statistics are provided for the previous hour, day, and week. Because each storage account is mapped to a shared folder (and the top-level sub folders of a shared folder are mapped to containers) you will see non-zero counts of those statistics before you access data using the object storage protocols. The GET, HEAD, OPTION, POST, PUT, and DELETE are HTTP methods used in the object storage API, and the dashboard shows the usage count for each method as well as the number of bytes transferred.

User management

To allow a user to access data by object storage protocols, press the [Add] button to choose users from the Turbo NAS in the user management page. When choosing Turbo NAS users, you can choose to add domain users if your Turbo NAS is configured with account authentication by AD or LDAP. You can also remove any users from the list.

After you add a user, they can access the object storage services using their username and password. However, using access keys is more secure.

Please note: After you delete a Turbo NAS user using the QTS Control Panel, the user and their access keys will remain valid. You must also remove the user from the OSS user list.

Access key management

You can create access keys for users once they are added to the OSS user list. You will use the access keys in applications to access the data by object storage protocols, and the application will access the data on behalf of the owner of the access key.

OSS provides multiple access keys for a user. If any access keys are compromised, you can remove the access keys without any impact of other applications that use different access keys. It is a common security practice to periodically replace old access keys with new ones.

Click on the [Property] button of an access key to view the details. Because of the different protocols, the service endpoint and terminology of S3 and OpenStack is different. For OpenStack, the service endpoint is the URL of its authentication service, and for S3, the service endpoint is the URL of its object storage service. To access data via OpenStack protocol, you use a username and API key, and for using S3 protocol, you use an access key and secret key.

Because of the multi-tenancy nature of object storage, you need to include the storage account in your OpenStack username and in your S3 access key. For example, if your OSS key ID is “bB9MSEIrXEYFGcxXdW7f” and the storage account you would like to access is “cloudvault”, your OpenStack username or the S3 access key will be “cloudvault:bB9MSEIrXEYFGcxXdW7f”. Please note that this can be used to access only the “cloudvault” storage account.

Storage account management

Click on the [Storage Account] item in the left-hand menu of OSS to manage storage accounts. Because storage accounts are equivalent to shared folders, you will see your shared folders are listed as storage accounts. You may also create a storage account here by pressing the [Create] button, and a corresponding shared folder will be created with default shared folder settings. You can also delete storage accounts here by selecting storage accounts and pressing the [Delete] button, which will also delete the corresponding shared folders.

Please note: There are some default shared folders (including home, homes, USB, and TMBackup) that are not accessible as storage accounts. The object storage statistics will also exclude these shared folders.

To configure the access permissions for a storage account, press its [Permission] icon in the Action column, and you can add users to the list. The only users that can be added are the users that have been added to the OSS user list. After you allow a user to access a storage account, all of the access keys owned by the user can be used to access this storage account.

The object storage protocols provide an API to retrieve the usage of storage accounts and containers. Because it takes time and system resources to calculate the statistics, the counting is done periodically. You can choose the time/date of when these are calculated.

Container management

You can click on the [Container] item in the left-hand menu of OSS to manage the containers. A container is a top-level folder in a shared folder. You can click on a storage account to list all its containers, and also create or delete containers.

Please note: There are also some top-level directories in the shared folder not accessible as containers: @Recycle, tmp, async_pending, and hidden folders. The object storage statistics will also exclude these top-level folders.

A public container allows anonymous access to all of the objects in it. This is useful when you want anyone to download a lot of files, or host web sites using the files in the container. To do so, choose the container and change its permission from Private to Public by pressing the [Permission] button. The URL for the public container will look like “http://nas_ip_address:oss_service_port/v1/AUTH_storage_account_name/file_path”.

Service endpoints

OSS offers data access services with a built-in server and also by the QTS web server using a virtual host. Using the OSS built-in server, you can configure its port. However it does not support HTTPS. If using the QTS web server, you need to configure the virtual host name, and the ports of your object storage services will be the same as your QTS web server. It can also support HTTPS. If you are using a virtual host name, your object storage client applications can only access the object storage service by FQDN.


Reporting issues

OSS App internally generates debug logs for facilitating troubleshooting. Please click on the [i] button on the top-right corner of the OSS window. A small window will show with the version information as well as a [Generate debug report] button. Press the button to download the debug log file for sending to the QNAP customer support team.

Data protection

The OSS App does not replicate objects, but you can protect the data by storing the objects in a RAID-protected volume. RAID offers better performance and storage efficiency than replication-based protection. For disaster recovery, you can use RTRR or rsync to replicate all of your objects to a remote Turbo NAS or CloudBackup NAS Apps to back up your objects to public cloud services. For data security, you can use the volume encryption provided by QTS to encrypt your objects.

Accessing object storage

Using CloudBerry Explorer

Cloudberry Lab offers a series of desktop applications for accessing and managing data in a variety of cloud storage environments. Please see below for setting up connections to OSS service. You can visit http://www.cloudberrylab.com/ for more information.

S3 Explorer

  1. Click on [File]->[S3 Compatible]->[S3-Compatible] to open the connection setup window.
  2. Enter the IP address or FQDN of your Turbo NAS and the port number of the object storage service in the “Service point” field, and also the access key and secret. If you use a virtual host, you will need to configure the DNS or modify your computer’s hosts file to map the virtual host name to the IP address of your Turbo NAS. If you choose to use SSL to encrypt the data transfer, you will get a warning message if your Turbo NAS uses a self-signed SSL certificate.
  3. Press [Test Connection] button to ensure correct settings, and select [OK] button to close the window. You can now start using it to access your data.

OpenStack Storage Explorer

  1. Click on [File]->[OpenStack] to open the connection setup window.
  2. Enter the URL of your OSS OpenStack service endpoint, with the proper IP address or FQDN of your Turbo NAS and the port number of object storage service in the “Authentication Service” field, and also the username and API key. If you use a virtual host, you will need to configure the DNS or modify your computer’s hosts file to map the virtual host name to the IP address of your Turbo NAS. You can choose to use SSL to encrypt the data transfer by using HTTPS, but you will get a warning message if your Turbo NAS uses a self-signed SSL certificate. Do NOT enable “Use Keystone authentication”.
  3. Press [Test Connection] button to ensure correct settings, and select the [OK] button to close the window. You can now start using it to access your data.

Using QNAP CloudBackup Apps

QNAP offers CloudBackup Apps that support backing up data from your Turbo NAS to a variety of cloud storage services including S3 and OpenStack. Please go to the QTS App Center to download these apps and follow their application notes to use object storage protocols as your backup solutions. You can now have one app on your Turbo NAS to backup to public cloud storage and private cloud storage.

More software utilities

There are many applications for S3 and OpenStack. Some are listed below for your reference:

  • Cyberduck (https://cyberduck.io/): S3 and OpenStack browser for Windows and Mac
  • Duplicati (http://www.duplicati.com/): S3 and OpenStack backup client on Windows and Linux
  • S3 Browser (http://s3browser.com/): S3 Windows client
  • S3cmd (http://s3tools.org/): Command Line Client Software for S3 in Linux
  • Cloud Explorer (https://github.com/rusher81572/cloudExplorer): An S3 browser for Windows, Mac, and Linux
  • s3fs (https://code.google.com/p/s3fs/): A FUSE-based file system backed by S3. Mount a bucket as a local file system read/write.

Developer kits

You can also find more open-source resources (SDK) via the Internet for developing S3 and OpenStack applications.

  • AWS sample code, libraries, and documents (http://aws.amazon.com/code)
  • OpenStack software development kits (https://wiki.openstack.org/wiki/SDKs)

Performance evaluation

You can test the performance of the object storage services for your Turbo NAS using the following tools:

  • COSBench (https://github.com/intel-cloud/cosbench)
  • Swift Bench (https://github.com/openstack/swift-bench)

Supported APIs

OSS supports the basic feature set of S3 and OpenStack API. However, because OSS enforces unified storage allowing data access using both objects and files, the behaviors of the API are slightly different from pure object storage. Please conduct testing before you use OSS for your production services.

OpenStack storage

  • Get Account Metadata
  • Swift ACLs
  • List Containers
  • Delete Container
  • Create Container
  • Get Container Metadata
  • Update Container Metadata
  • Delete Container Metadata
  • List Objects
  • Create Object (max object size: 1TB)
  • Create Large Object (support SLO and DLO)
  • Recommend segment size: 5GB
  • Delete Object (support bulk delete)
  • Get Object
  • Copy Object
  • Get Object Metadata
  • Update Object Metadata
  • CORS
  • TempURL
  • Form POST

S3

  • List Buckets
  • Delete Bucket
  • Create Bucket
  • Get Bucket Info (HEAD)
  • Put Object
  • Delete Object
  • Delete Multiple Object
  • Get Object
  • Get Object Info (HEAD)
  • Copy Object
  • Multipart Uploads
Datum vydání: 2014-11-28
Bylo to pro vás to užitečné?
Děkujeme vám za vaši zpětnou vazbu.
Děkujeme vám za vaši zpětnou vazbu. Máte-li nějaké dotazy, kontaktujte prosím adresu support@qnap.com
100% lidí si myslí, že to pomáhá.