Difference between revisions of "ARK2/Files"
(→File Management) |
(→File Management) |
||
| Line 8: | Line 8: | ||
* Image management uses [https://github.com/avalanche123/Imagine Imagine]. | * Image management uses [https://github.com/avalanche123/Imagine Imagine]. | ||
* Thumbnails are stored separately from the master files to make backups, archiving, and regeneration easier | * Thumbnails are stored separately from the master files to make backups, archiving, and regeneration easier | ||
| + | * Multiple thumbnail profiles can be configured | ||
| + | * Thumbnails will be generated asynchronously | ||
The Media Types and File Types supported are configurable, with standard types provided by default: | The Media Types and File Types supported are configurable, with standard types provided by default: | ||
| − | * Image | + | * Image (jpg, png, etc) |
| − | + | * Audio (mp3, etc) | |
| − | * Audio | + | * Video (mkv, etc) |
| − | + | * Text (txt, etc) | |
| − | * Video | + | * Document / Application (pdf, odt, ods, etc) |
| − | + | * Spatial? | |
| − | * Text | ||
| − | |||
| − | * Document / Application | ||
| − | |||
* Other | * Other | ||
| − | |||
| − | |||
| − | |||
In multi-tenant mode, the files for each ARK instance are stored separately under the 'sites' folder, e.g. 'sites/mysite/files'. | In multi-tenant mode, the files for each ARK instance are stored separately under the 'sites' folder, e.g. 'sites/mysite/files'. | ||
| Line 44: | Line 39: | ||
The individual file names will be in a standard format. Flysystem abstracts folders and file names as a relative path, which will take the form: | The individual file names will be in a standard format. Flysystem abstracts folders and file names as a relative path, which will take the form: | ||
files/data/<mediatype>/<token>/<id>.<revision>.<suffix> | files/data/<mediatype>/<token>/<id>.<revision>.<suffix> | ||
| + | files/thumbs/<mediatype>/<token>/<id>.<revision>.<profile>.<suffix> | ||
== Design Work == | == Design Work == | ||
| Line 60: | Line 56: | ||
* Security | * Security | ||
| − | + | Problems with current setup: | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | Problems with current | ||
* All files in one directory, can be slow for large volumes | * All files in one directory, can be slow for large volumes | ||
* Thumbnails in same directory, slows performance, harder to maintain | * Thumbnails in same directory, slows performance, harder to maintain | ||
Revision as of 18:47, 16 November 2016
File Management
File and Media management have seen major changes in ARK2, with a more flexible system implemented.
- Files are now a core Module instead of a dataclass and look-up table, allowing for flexible metadata based on file type, and using files as either data properties or relationships.
- The file system is abstracted using FlySystem allowing files to be transparently saved locally or to cloud providers.
- File versioning is supported.
- Image management uses Imagine.
- Thumbnails are stored separately from the master files to make backups, archiving, and regeneration easier
- Multiple thumbnail profiles can be configured
- Thumbnails will be generated asynchronously
The Media Types and File Types supported are configurable, with standard types provided by default:
- Image (jpg, png, etc)
- Audio (mp3, etc)
- Video (mkv, etc)
- Text (txt, etc)
- Document / Application (pdf, odt, ods, etc)
- Spatial?
- Other
In multi-tenant mode, the files for each ARK instance are stored separately under the 'sites' folder, e.g. 'sites/mysite/files'.
The directory structure for each instance is as follows:
- files
|- download
|- upload
|- tmp
|- data
|- <mediatype>
|- <tokens>
|- thumbs
|- <mediatype>
|- <tokens>
where the <mediattype> is as defined in the configuration, and the <token> is a configurable token to split the folders into manageable sizes. By default the <token> will be split by file id in groups of 1000, so subfolders for 0, 1000, 2000, 3000, etc.
The individual file names will be in a standard format. Flysystem abstracts folders and file names as a relative path, which will take the form:
files/data/<mediatype>/<token>/<id>.<revision>.<suffix> files/thumbs/<mediatype>/<token>/<id>.<revision>.<profile>.<suffix>
Design Work
File Manage needs to be flexible and fast.
- File attachments to data items
- Data downloads / exports
- Documentation
- Temp files
- Mapping files
- Image management, generated thumbnails, etc
- Metadata, mimetypes, etc
- Cloud storage
- Efficiency
- Security
Problems with current setup:
- All files in one directory, can be slow for large volumes
- Thumbnails in same directory, slows performance, harder to maintain
- No versioning
- No expiry
Full document management and versioning workflow is a stretch goal needed for Avalon. Try use CMIS standard as used in LibreOffice.
Need to plan for moving to advanced model while keeping simple for first release.
- Files are a system module, with modtype for core file type and attributes as required
- Required module properties for title and description, all else optional, some core fields replicated on item table to performance
- Use FlySystem for file system abstraction, extend to allow subpaths to be transparently mounted on.
- Use Imagine for image handling
- Split files into subdirs with max number of files per dir
- Thumbnails into separate dir, managed by image code (on-the-fly creation, etc)
- Support versioning
- Support expiry date for tmp files
- Support mimetypes + core types (image, video, audio, document, text)
Very similar to http://documentation.concrete5.org/developers/working-with-files-and-the-file-manager/overview
