Metadata Acceleration Bucket reuses the COS Storage Bucket Inventory feature, providing file metadata export functionality similar to the Hadoop OIV tool. The metadata acceleration bucket can perform scheduled scans of files in the specified path within the user's bucket daily or weekly according to the user's inventory task configuration, and produce an inventory report stored in CSV format in the user-specified path. It is suitable for file analysis, audit, governance, and other scenarios. Note:
The Metadata Acceleration Bucket Inventory Feature allowlist is currently in public beta. This document is for reference during the beta test. To apply for enabling the inventory feature, please contact us. During the public beta period, the inventory feature is temporarily free of COS management feature fees. Subsequent pricing information will be notified via internal message and official documentation. Use Limits
Metadata acceleration bucket inventory has the following limitations during the public beta period:
Each COS metadata acceleration bucket supports a maximum of 5 configured inventory rules.
Note:
Typically, it is recommended to configure the bucket root path as the manifest scan path. If your usage scenario requires configuring more than 5 rules, please contact us. Inventory reports do not support cross-bucket delivery. The target path in task configuration must match the source bucket.
Inventory does not support custom IDs. A task ID is automatically assigned upon task creation. This ID can be queried via the List API.
Metadata acceleration buckets do not currently support instant inventory.
Inventory Report Delivery Instructions
After configuring an inventory task, COS will perform scheduled scans of files in the specified path of your bucket based on the configuration and output an inventory report in CSV format. The delivery path for the report can be specified in the task configuration. The specific path for the CSV file is as follows:
destination-prefix/YYYY-MM-DD-HH/Data/8d1048acf48096d17502bd62efc9f8c1.csv
destination-prefix: The shipping route specified by the user in the list delivery task.
YYYY-MM-DD-HH: Timestamp, the year, month, day, and hour when the inventory task is executed.
Note:
The hour in the path indicates the task start execution time, which may differ from the actual delivery time. COS metadata acceleration buckets can only be configured for daily tasks. The initial configuration or delivery within 24 hours will generate one copy of the inventory report. It is unable to specify a fixed time to execute inventory tasks or deliver inventory reports.
Inventory report: Delivered in CSV format, with filenames as random values generated by COS.
Manifest tasks scan files under the configured path according to rule configuration. The generated report excludes header info and prints field order as follows:
|
FileName | File name. |
ATime | File access time Note: To ensure the best read/write performance, the metadata acceleration bucket does not enable ATime tracing by default. If you need the ATime feature, contact us. |
MTime | Change time of file content |
CTime | Change time of file metadata |
Size | File size (Byte) |
FileType | Print file status as directory or file, output format: DIR (directory) or FILE (file) |
ACL | Print file permissions in digits, for example: 755 |
UserName | Username, for example: hadoop |
GroupName | User group name, for example: supergroup |
StorageClass | The current file storage type. Possible values: STANDARD (standard storage) DEGRADE (infrequent storage) ARCHIVE (archive storage) DEEP_ARCHIVE (deep archive storage) ARCHIVE+STANDARD (archive storage restored to standard) DEEP_ARCHIVE+STANDARD (deep archive storage restored to standard) |
Manifest Configuration Method
Currently, metadata acceleration buckets in COS cannot be accessed through the console for inventory configuration and only supports using APIs to configure. Specific steps are as follows:
1. Create a COS role.
2. Bind permissions to the COS role.
3. Enable the inventory feature.
1. Creating a COS role
Create a COS role. For more information about the API, see CreateRole. Here, roleName should be COS_QcsRole.
policyDocument should be:
{
"version": "2.0",
"statement": [{
"action": "name/sts:AssumeRole",
"effect": "allow",
"principal": {
"service": "cos.cloud.tencent.com"
}
}]
}
2. Binding Permissions to the COS Role
Bind permissions to the role. For more information about the API, see AttachRolePolicy. Here, policyName is QcloudCOSFullAccess, and roleName is the COS_QcsRole in step 1 or the roleID returned when roleName is created.
3. Enabling the Inventory Feature
Call the API to enable the inventory feature. Metadata acceleration buckets reuse the COS inventory capacity and support using the COS Inventory API to enable inventory configuration. Due to the metadata acceleration bucket feature, some fields in the API differ from the COS inventory API. For the specific calling method, please refer to the following API example:
This interface is used to create inventory tasks in the bucket. For the interface parameter description, refer to COS PUT Inventory API Documentation. Due to the metadata acceleration bucket feature, some fields in the API differ from the COS inventory interface. Details are as follows: Metadata acceleration bucket inventory does not support specifying an ID. When creating a task, ID information is not needed, as the ID is automatically generated by COS.
Metadata acceleration buckets currently only support delivering inventory reports to the source bucket. COSBucketDestination needs to be configured as the source bucket.
Inventory report encryption cannot be configured temporarily. The Encryption field cannot be configured when creating a task.
Metadata acceleration buckets do not support version control capability, so the IncludedObjectVersions field is unsupported during configuration. Metadata Bucket cannot be specified to generate inventory report fields. For the OptionalFields field, refer to the example below for input parameters.
Request
PUT /?inventory HTTP/1.1
Host: <BucketName-APPID>.cos.<Region>.myqcloud.com
Date: GMT Date
Authorization: Auth String
Content-MD5: MD5
<InventoryConfiguration>
<IsEnabled>true</IsEnabled>
<Destination>
<COSBucketDestination>
<Format>CSV</Format>
<AccountId>1000000000</AccountId>
<Bucket>qcs::cos:ap-beijing::bucketname-100000000</Bucket>
<Prefix>/testprefix</Prefix>
</COSBucketDestination>
</Destination>
<Schedule>
<Frequency>Daily</Frequency>
</Schedule>
<Filter>
<Prefix>/my-inventory</Prefix>
</Filter>
<OptionalFields>
<Field>ATime</Field>
<Field>ACL</Field>
<Field>MTime</Field>
<Field>CTime</Field>
<Field>Size</Field>
<Field>UserName</Field>
<Field>GroupName</Field>
<Field>StorageClass</Field>
<Field>FileType</Field>
</OptionalFields>
</InventoryConfiguration>
Response
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 0
Date: Mon, 28 Aug 2018 02:53:38 GMT
Server: tencent-cos
x-cos-request-id: NTlhMzg1ZWVfMjQ4OGY3MGFfMWE1NF8****
This interface is used to query specific inventory task information in the bucket. For the interface parameter description, refer to COS GET Inventory API Documentation. This interface requires the inventory ID parameter, which is automatically generated by the metadata acceleration bucket when creating a task. If needed, call the List Bucket Inventory Configurations API to obtain this ID. Request
GET /?inventory&id=inventory-configuration-ID HTTP/1.1
Host: <BucketName-APPID>.cos.<Region>.myqcloud.com
Date: GMT Date
Authorization: Auth String
Response
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 331
Date: Mon, 28 Aug 2018 02:53:39 GMT
Server: tencent-cos
x-cos-request-id: NTlhMzg1ZWVfMjQ4OGY3MGFfMWE1NF84Y2M
<?xml version = "1.0" encoding = "UTF-8">
<InventoryConfiguration xmlns = "http://....">
<Id>12761</Id>
<IsEnabled>true</IsEnabled>
<Destination>
<COSBucketDestination>
<Format>CSV</Format>
<AccountId>1000000000</AccountId>
<Bucket>qcs::cos:ap-beijing::bucketname-100000000</Bucket>
<Prefix>/testprefix</Prefix>
</COSBucketDestination>
</Destination>
<Schedule>
<Frequency>Daily</Frequency>
</Schedule>
<Filter>
<Prefix>/my-inventory</Prefix>
</Filter>
<OptionalFields>
<Field>ATime</Field>
<Field>ACL</Field>
<Field>MTime</Field>
<Field>CTime</Field>
<Field>Size</Field>
<Field>UserName</Field>
<Field>GroupName</Field>
<Field>StorageClass</Field>
<Field>FileType</Field>
</OptionalFields>
</InventoryConfiguration>
List Bucket Inventory Configurations
Request
GET /?inventory HTTP/1.1
Host: <BucketName-APPID>.cos.<Region>.myqcloud.com
Date: GMT Date
Authorization: Auth String
Response
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 331
Date: Mon, 28 Aug 2018 02:53:39 GMT
Server: tencent-cos
x-cos-request-id: NTlhMzg1ZWVfMjQ4OGY3MGFfMWE1NF8****
<?xml version = "1.0" encoding = "UTF-8">
<ListInventoryConfigurationResult xmlns = "http://....">
<InventoryConfiguration>
<Id>12761</Id>
<IsEnabled>true</IsEnabled>
<Destination>
<COSBucketDestination>
<Format>CSV</Format>
<AccountId>1000000000</AccountId>
<Bucket>qcs::cos:ap-beijing::bucketname-100000000</Bucket>
<Prefix>/testprefix</Prefix>
</COSBucketDestination>
</Destination>
<Schedule>
<Frequency>Daily</Frequency>
</Schedule>
<Filter>
<Prefix>/my-inventory</Prefix>
</Filter>
<OptionalFields>
<Field>ATime</Field>
<Field>ACL</Field>
<Field>MTime</Field>
<Field>CTime</Field>
<Field>Size</Field>
<Field>UserName</Field>
<Field>GroupName</Field>
<Field>StorageClass</Field>
<Field>FileType</Field>
</OptionalFields>
</InventoryConfiguration>
<InventoryConfiguration>
<Id>12762</Id>
<IsEnabled>true</IsEnabled>
<Destination>
<COSBucketDestination>
<Format>CSV</Format>
<AccountId>1000000000</AccountId>
<Bucket>qcs::cos:ap-beijing::bucketname-100000000</Bucket>
<Prefix>/testprefix</Prefix>
</COSBucketDestination>
</Destination>
<Schedule>
<Frequency>Daily</Frequency>
</Schedule>
<Filter>
<Prefix>/my-inventory2</Prefix>
</Filter>
<OptionalFields>
<Field>ATime</Field>
<Field>ACL</Field>
<Field>MTime</Field>
<Field>CTime</Field>
<Field>Size</Field>
<Field>UserName</Field>
<Field>GroupName</Field>
<Field>StorageClass</Field>
<Field>FileType</Field>
</OptionalFields>
</InventoryConfiguration>
<IsTruncated>false</IsTruncated>
</ListInventoryConfigurationResult>
This interface is used to delete specified inventory tasks in a bucket. For the interface parameter description, refer to COS DELETE Bucket Inventory API Documentation. This API requires the inventory ID parameter, which is automatically generated by the metadata acceleration bucket when creating a task. If needed, call the List Bucket Inventory Configurations API to obtain this ID. Request
DELETE /?inventory&id=inventory-configuration-id HTTP/1.1
Host: <BucketName-APPID>.cos.<Region>.myqcloud.com
Date: GMT Date
Authorization: Auth String
Response
HTTP/1.1 204 No Content
Server: tencent-cos
Date: Mon, 28 Aug 2018 02:53:40 GMT
x-cos-id-2:0dfafa/DAPDIFdafdsfDdfSFFfdfKKJdafasiuKJK2
x-cos-request-id: NTlhM2I3M2JfMjQ4OGY3MGFfMWE1NF84****